Nov 02 21:07:51.466457 ip-10-0-133-86 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 21:07:51.914795 ip-10-0-133-86 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:07:51.914795 ip-10-0-133-86 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 21:07:51.914795 ip-10-0-133-86 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:07:51.916783 ip-10-0-133-86 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:07:51.924122 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924028    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924125    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924131    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924135    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924139    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924142    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924147    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924152    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924156    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924160    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924163    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924165    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924168    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924171    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924173    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924176    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:07:51.924172 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924179    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924182    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924185    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924187    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924190    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924193    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924196    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924199    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924201    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924203    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924206    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924209    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924211    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924214    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924217    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924220    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924222    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924224    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924227    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:07:51.924578 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924230    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924232    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924235    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924237    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924240    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924244    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924247    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924250    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924254    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924256    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924259    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924262    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924264    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924267    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924270    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924273    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924275    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924278    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924281    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:07:51.925012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924283    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924286    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924288    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924291    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924293    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924295    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924298    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924300    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924303    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924305    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924307    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924310    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924312    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924315    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924317    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924320    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924322    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924325    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924328    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924331    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:07:51.925469 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924333    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924336    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924338    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924341    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924344    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924346    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924349    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924362    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924366    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924368    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924371    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924373    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924376    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924379    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924381    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924384    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924386    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924408    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924411    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:07:51.925940 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924414    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924417    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924419    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924422    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924425    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924427    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924430    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924432    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924435    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924438    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924441    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924443    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.924446    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925325    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925333    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925337    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925340    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925344    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925346    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925349    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:07:51.926391 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925352    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925356    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925359    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925361    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925364    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925367    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925370    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925372    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925375    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925378    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925380    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925383    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925385    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925388    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925391    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925393    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925396    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925398    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925402    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925405    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:07:51.926856 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925408    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925410    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925413    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925415    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925418    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925421    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925424    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925426    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925429    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925432    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925434    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925437    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925440    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925442    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925445    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925448    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925451    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925454    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925457    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925460    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:07:51.927330 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925463    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925465    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925468    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925471    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925473    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925476    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925479    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925481    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925484    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925486    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925489    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925492    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925494    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925498    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925502    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925505    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925509    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925512    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925515    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:07:51.927879 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925518    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925520    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925523    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925526    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925528    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925531    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925533    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925536    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925540    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925543    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925546    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925548    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925551    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925554    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925556    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925559    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925561    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925564    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925567    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925569    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:07:51.928332 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925572    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925574    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925577    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925579    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925582    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925585    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925587    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925590    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925592    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925595    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925598    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925600    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925604    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925618    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925621    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925623    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925626    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925628    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925631    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:07:51.928828 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.925633    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926140    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926145    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926149    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926152    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926155    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926157    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926160    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926162    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926165    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926167    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926170    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926172    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926175    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926177    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926180    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926182    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926185    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926188    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:07:51.929291 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926190    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926193    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926196    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926198    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926201    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926203    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926206    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926208    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926210    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926213    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926215    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926218    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926221    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926224    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926227    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926230    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926233    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926236    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926238    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:07:51.929748 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926241    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926244    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926248    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926250    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926253    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926256    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926258    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926261    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926263    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926266    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926268    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926271    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926273    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926275    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926278    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926281    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926283    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926286    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926289    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:07:51.930222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926292    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926295    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926297    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926300    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926302    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926304    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926307    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926309    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926312    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926314    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926317    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926320    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926323    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926325    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926328    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926330    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926333    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926335    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926338    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926340    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:07:51.930662 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926342    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926345    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926347    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926350    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926353    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926355    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926357    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926360    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926363    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926365    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926368    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926370    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926373    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926375    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926377    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926380    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926382    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926385    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926387    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926390    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:07:51.931126 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926392    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926395    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926398    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926401    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926403    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926406    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926408    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926412    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926414    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.926417    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926508    2583 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926515    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926521    2583 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926526    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926530    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926533    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926537    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926542    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926545    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926548    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926551    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 21:07:51.931627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926555    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926558    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926563    2583 flags.go:64] FLAG: --cgroup-root=""
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926566    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926569    2583 flags.go:64] FLAG: --client-ca-file=""
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926572    2583 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926575    2583 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926580    2583 flags.go:64] FLAG: --cluster-domain=""
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926583    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926586    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926589    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926592    2583 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926596    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926599    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926602    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926606    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926609    2583 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926612    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926615    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926618    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926620    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926625    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926627    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926631    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926633    2583 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 21:07:51.932092 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926636    2583 flags.go:64] FLAG: --enable-server="true"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926639    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926643    2583 flags.go:64] FLAG: --event-burst="100"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926646    2583 flags.go:64] FLAG: --event-qps="50"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926649    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926654    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926657    2583 flags.go:64] FLAG: --eviction-hard=""
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926661    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926664    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926670    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926673    2583 flags.go:64] FLAG: --eviction-soft=""
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926676    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926679    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926682    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926685    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926687    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926690    2583 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926693    2583 flags.go:64] FLAG: --feature-gates=""
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926697    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926700    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926703    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926706    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926709    2583 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926712    2583 flags.go:64] FLAG: --help="false"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926716    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:51.932757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926719    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926722    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926725    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926728    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926732    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926735    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926737    2583 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926740    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926743    2583 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926746    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926749    2583 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926752    2583 flags.go:64] FLAG: --kube-reserved=""
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926756    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926758    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926762    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926764    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926768    2583 flags.go:64] FLAG: --lock-file=""
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926771    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926774    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926777    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926783    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926785    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926788    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 21:07:51.933371 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926791    2583 flags.go:64] FLAG: --logging-format="text"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926794    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926797    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926800    2583 flags.go:64] FLAG: --manifest-url=""
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926803    2583 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926807    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926810    2583 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926815    2583 flags.go:64] FLAG: --max-pods="110"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926818    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926821    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926825    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926827    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926830    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926833    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926836    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926843    2583 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926846    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926849    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926852    2583 flags.go:64] FLAG: --pod-cidr=""
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926855    2583 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926858    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926862    2583 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926865    2583 flags.go:64] FLAG: --port="10250"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926869    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926871    2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0004c62d18ace5d95"
Nov 02 21:07:51.933926 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926876    2583 flags.go:64] FLAG: --qos-reserved=""
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926879    2583 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926882    2583 flags.go:64] FLAG: --register-node="true"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926885    2583 flags.go:64] FLAG: --register-with-taints=""
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926889    2583 flags.go:64] FLAG: --registry-burst="10"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926892    2583 flags.go:64] FLAG: --registry-qps="5"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926895    2583 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926898    2583 flags.go:64] FLAG: --reserved-memory=""
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926901    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926905    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926907    2583 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926911    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926914    2583 flags.go:64] FLAG: --runonce="false"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926917    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926920    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926923    2583 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926927    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926930    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926933    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926936    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926939    2583 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926942    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926945    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926948    2583 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926950    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926954    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 21:07:51.934538 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926957    2583 flags.go:64] FLAG: --system-cgroups=""
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926959    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926966    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926969    2583 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926972    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926977    2583 flags.go:64] FLAG: --tls-min-version=""
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926981    2583 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926984    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926987    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926990    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926993    2583 flags.go:64] FLAG: --v="2"
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.926997    2583 flags.go:64] FLAG: --version="false"
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.927001    2583 flags.go:64] FLAG: --vmodule=""
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.927006    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.927009    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927156    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927160    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927163    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927165    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927168    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927171    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927174    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927176    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:07:51.935185 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927180    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927182    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927185    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927187    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927190    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927192    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927195    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927197    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927200    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927202    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927205    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927209    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927212    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927214    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927217    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927220    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927223    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927226    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927228    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927230    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:07:51.936733 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927234    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927236    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927239    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927241    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927244    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927246    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927249    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927251    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927254    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927256    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927259    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927261    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927264    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927267    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927270    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927272    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927275    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927277    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927280    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927282    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:07:51.937251 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927285    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927287    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927290    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927293    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927296    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927298    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927301    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927304    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927307    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927310    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927312    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927315    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927318    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927320    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927323    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927325    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927328    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927330    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927333    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927335    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:07:51.937736 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927338    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927341    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927343    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927346    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927349    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927351    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927354    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927357    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927359    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927362    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927365    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927367    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927370    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927372    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927374    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927378    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927381    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927383    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927386    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:07:51.938210 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927390    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927392    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927395    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927397    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927400    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927402    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927405    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927407    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927410    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927412    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927415    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927417    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927420    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927424    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927428    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927431    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927434    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927439    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:07:51.938646 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927441    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.927468    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927506    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927510    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927513    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927516    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927518    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927521    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927523    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927526    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927530    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927532    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927535    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927537    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927540    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927543    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:07:51.939062 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927546    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927548    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927551    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927553    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927556    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927559    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927561    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927564    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927567    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927569    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927571    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927574    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927577    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927579    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927582    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927584    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927587    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927589    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927592    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927596    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:07:51.939534 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927598    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927601    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927603    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927606    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927609    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927611    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927615    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927617    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927620    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927623    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927625    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927629    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927631    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927634    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927636    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927639    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927641    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927644    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927646    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927649    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:07:51.939985 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927651    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927654    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927656    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927658    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927661    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927663    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927666    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927668    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927671    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927673    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927676    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927678    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927681    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927684    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927686    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927689    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927692    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927694    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927698    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:07:51.940507 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927700    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927703    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927705    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927708    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927712    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927715    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927718    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927720    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927723    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927725    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927728    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927730    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927733    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927736    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927738    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927741    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927743    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927746    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927748    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:07:51.940955 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927751    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927754    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927758    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927761    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927765    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927769    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927772    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927776    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927779    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927781    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927784    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927787    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927791    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.927795    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.935907    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.935922    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.936640    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.936647    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936707    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:07:51.941430 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936712    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936716    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936719    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936722    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936725    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936727    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936730    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936733    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936736    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936739    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936742    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936745    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936748    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936750    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936754    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936756    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936759    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936762    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936764    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936767    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:07:51.941981 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936769    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936772    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936775    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936777    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936780    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936783    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936786    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936789    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936792    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936795    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936798    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936800    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936803    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936806    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936808    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936811    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936814    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936818    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936821    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:07:51.942482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936825    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936828    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936831    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936834    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936838    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936841    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936844    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936847    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936849    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936852    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936854    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936857    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936859    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936862    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936866    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936869    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936872    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936874    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936877    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:07:51.942944 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936880    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936883    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936886    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936890    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936892    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936895    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936898    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936901    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936903    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936906    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936909    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936911    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936914    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936917    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936920    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936922    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936925    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936927    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936930    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936932    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:07:51.943429 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936935    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936938    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936940    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936943    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936945    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936948    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936951    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936953    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936956    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936958    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936961    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936964    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936967    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936970    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936972    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936975    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936978    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936981    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936983    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936986    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:07:51.943883 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936988    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936991    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936993    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936996    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.936998    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937001    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937003    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.937032    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937190    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937195    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937198    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937201    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937203    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937207    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937211    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937214    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:07:51.944555 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937217    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937220    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937224    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937227    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937230    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937234    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937236    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937239    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937242    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937245    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937247    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937250    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937253    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937256    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937259    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937262    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937265    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937267    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937270    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:07:51.945239 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937272    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937275    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937277    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937280    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937282    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937285    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937287    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937290    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937293    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937296    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937298    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937301    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937303    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937306    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937309    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937311    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937314    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937316    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937319    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937321    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:07:51.945847 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937324    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937326    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937329    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937332    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937334    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937337    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937339    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937343    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937345    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937348    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937350    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937353    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937355    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937358    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937360    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937363    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937365    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937368    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937370    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937373    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:07:51.946352 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937375    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937378    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937381    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937384    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937386    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937389    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937392    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937394    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937396    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937399    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937401    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937404    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937406    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937409    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937412    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937414    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937417    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937419    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937421    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:07:51.947012 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937424    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937427    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937429    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937432    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937435    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937437    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937439    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937442    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937444    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937447    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937450    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937452    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937455    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937458    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937460    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937463    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937465    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937468    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937470    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:07:51.947463 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937473    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.937496    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937534    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937538    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937541    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937544    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937547    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937549    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937552    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937554    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937557    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937560    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937562    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937565    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937567    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937574    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:07:51.947886 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937578    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937580    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937583    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937585    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937588    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937590    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937593    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937596    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937598    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937614    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937616    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937619    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937621    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937623    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937626    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937628    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937630    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937633    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937636    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937638    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:07:51.948311 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937641    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937643    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937646    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937648    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937650    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937653    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937655    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937658    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937663    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937666    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937668    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937671    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937674    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937676    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937679    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937681    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937684    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937687    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937689    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:07:51.948867 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937692    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937694    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937697    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937699    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937702    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937705    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937707    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937710    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937713    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937715    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937718    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937721    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937723    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937726    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937729    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937731    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937733    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937736    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937738    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:07:51.949351 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937742    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937746    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937750    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937753    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937757    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937761    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937763    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937767    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937770    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937772    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937774    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937777    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937779    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937782    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937784    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937787    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937789    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937792    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937794    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:07:51.949824 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937796    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937800    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937803    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937805    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937808    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937810    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937812    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937815    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937817    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937820    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937822    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937826    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937828    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937831    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:51.937833    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.937891    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.940260    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.941364    2583 server.go:1014] "Starting client certificate rotation"
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.941474    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 21:07:51.950475 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.941523    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 21:07:51.950908 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.946968    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 21:07:51.950908 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.948371    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 21:07:51.961424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.961407    2583 log.go:25] "Validated CRI v1 runtime API"
Nov 02 21:07:51.965235 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.965220    2583 log.go:25] "Validated CRI v1 image API"
Nov 02 21:07:51.966230 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.966202    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 21:07:51.972040 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.972020    2583 fs.go:135] Filesystem UUIDs: map[3f364eca-ea43-4346-b859-733d9a639e1e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eeb97604-fe3d-4c55-b35f-937ee5c23d19:/dev/nvme0n1p3]
Nov 02 21:07:51.972127 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.972040    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 21:07:51.977142 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.977012    2583 manager.go:217] Machine: {Timestamp:2026-11-02 21:07:51.975714941 +0000 UTC m=+0.380740346 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3146222 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af6ffeba7bd69ee73c460b87dc911 SystemUUID:ec2af6ff-eba7-bd69-ee73-c460b87dc911 BootID:c8b54ec4-217c-4fc9-a6e4-8c8f904e4526 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:dd:75:ff:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:dd:75:ff:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:95:73:34:3d:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 21:07:51.977216 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.977144    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 21:07:51.977245 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.977224    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 21:07:51.981556 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.981528    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 21:07:51.981708 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.981558    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-86.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 21:07:51.981751 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.981711    2583 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 21:07:51.981751 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.981719    2583 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 21:07:51.981751 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.981735    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 21:07:51.982651 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.982640    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 21:07:51.983834 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.983824    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 21:07:51.988415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.988385    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 21:07:51.988543 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.988533    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 21:07:51.989389 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.989380    2583 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 21:07:51.989421 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.989393    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 21:07:51.989421 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.989404    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 21:07:51.989421 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.989413    2583 kubelet.go:405] "Adding apiserver pod source"
Nov 02 21:07:51.989421 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.989421    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 21:07:51.989838 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.989820    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 21:07:51.994366 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:51.994346    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 21:07:52.001532 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.001518    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 21:07:52.002452 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.002440    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 21:07:52.002497 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.002469    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 21:07:52.002616 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.002607    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 21:07:52.002649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.002622    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 21:07:52.002649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.002629    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 21:07:52.002649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.002634    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 21:07:52.002649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.002639    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 21:07:52.002649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.002645    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 21:07:52.003335 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.003327    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 21:07:52.003335 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.003336    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 21:07:52.003395 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.003343    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 21:07:52.003395 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.003349    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 21:07:52.003395 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.003358    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 21:07:52.003395 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.003367    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 21:07:52.004267 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.004258    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 21:07:52.004301 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.004268    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 21:07:52.008120 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.008093    2583 server.go:1263] "Started kubelet"
Nov 02 21:07:52.008229 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.008189    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 21:07:52.008270 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.008214    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 21:07:52.008299 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.008289    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 21:07:52.008891 ip-10-0-133-86 systemd[1]: Started Kubernetes Kubelet.
Nov 02 21:07:52.009878 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.009856    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 21:07:52.012368 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.012342    2583 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 21:07:52.015080 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.015062    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 21:07:52.020169 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.020141    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 21:07:52.020836 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.020819    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 21:07:52.020927 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.020914    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.021560 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.021540    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 21:07:52.021560 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.021542    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 21:07:52.021676 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.021590    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 21:07:52.021709 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.021697    2583 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 21:07:52.021709 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.021702    2583 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 21:07:52.021991 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.021976    2583 factory.go:55] Registering systemd factory
Nov 02 21:07:52.022066 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.021994    2583 factory.go:223] Registration of the systemd container factory successfully
Nov 02 21:07:52.022241 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.022231    2583 factory.go:153] Registering CRI-O factory
Nov 02 21:07:52.022329 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.022244    2583 factory.go:223] Registration of the crio container factory successfully
Nov 02 21:07:52.022329 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.022292    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 21:07:52.022329 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.022311    2583 factory.go:103] Registering Raw factory
Nov 02 21:07:52.022329 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.022326    2583 manager.go:1199] Started watching for new ooms in manager
Nov 02 21:07:52.023563 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.023497    2583 manager.go:319] Starting recovery of all containers
Nov 02 21:07:52.039130 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.035489    2583 manager.go:324] Recovery completed
Nov 02 21:07:52.041674 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.041658    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:07:52.043554 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.043530    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:07:52.043647 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.043573    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:07:52.043647 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.043584    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:07:52.044173 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.044161    2583 cpu_manager.go:226] "Starting" policy="none"
Nov 02 21:07:52.044173 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.044167    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:07:52.044173 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.044171    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 21:07:52.044327 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.044187    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 21:07:52.044920 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.044904    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:07:52.044999 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.044937    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:07:52.044999 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.044951    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:07:52.046394 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.046383    2583 policy_none.go:50] "Start"
Nov 02 21:07:52.046445 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.046397    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 21:07:52.046445 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.046407    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 21:07:52.048423 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.048412    2583 policy_none.go:44] "Start"
Nov 02 21:07:52.103842 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.098557    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 21:07:52.103842 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.099674    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 21:07:52.103842 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.099699    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 21:07:52.103842 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.099723    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 21:07:52.103842 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.099756    2583 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 21:07:52.121602 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.121573    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.137468 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.137430    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-86.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 21:07:52.137468 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.137430    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-86.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 21:07:52.138649 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.137431    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-86.us-west-2.compute.internal.18e45682f21dab68  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-86.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-86.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:07:52.008067944 +0000 UTC m=+0.413093350,LastTimestamp:2026-11-02 21:07:52.008067944 +0000 UTC m=+0.413093350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-86.us-west-2.compute.internal,}"
Nov 02 21:07:52.194357 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.194271    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-86.us-west-2.compute.internal.18e45682f43b28f6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-86.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-86.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-86.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:07:52.043555062 +0000 UTC m=+0.448580470,LastTimestamp:2026-11-02 21:07:52.043555062 +0000 UTC m=+0.448580470,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-86.us-west-2.compute.internal,}"
Nov 02 21:07:52.195055 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.195027    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 21:07:52.200635 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.200619    2583 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 21:07:52.222392 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.222373    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.224722 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.224695    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 21:07:52.227636 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.227623    2583 manager.go:341] "Starting Device Plugin manager"
Nov 02 21:07:52.227724 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.227701    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 21:07:52.227724 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.227712    2583 server.go:87] "Starting device plugin registration server"
Nov 02 21:07:52.227939 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.227928    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 21:07:52.227989 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.227940    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 21:07:52.228087 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.228072    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 21:07:52.228214 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.228162    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 21:07:52.228214 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.228171    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 21:07:52.228735 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.228710    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 21:07:52.228828 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.228772    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.267938 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.267916    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 21:07:52.268062 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.267960    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gdpv4"
Nov 02 21:07:52.283447 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.283407    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 02 21:07:52.283501 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.283389    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-86.us-west-2.compute.internal.18e45682f43b82a7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-86.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-86.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-86.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:07:52.043578023 +0000 UTC m=+0.448603427,LastTimestamp:2026-11-02 21:07:52.043578023 +0000 UTC m=+0.448603427,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-86.us-west-2.compute.internal,}"
Nov 02 21:07:52.283501 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.283483    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-86.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 21:07:52.283573 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.283501    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 21:07:52.328363 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.328327    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:07:52.329376 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.329360    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:07:52.329463 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.329393    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:07:52.329463 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.329402    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:07:52.329463 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.329429    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.382906 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.382875    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.382906 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.382908    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-86.us-west-2.compute.internal\": node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.400975 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.400949    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"]
Nov 02 21:07:52.401070 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.401012    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:07:52.402012 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.401992    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:07:52.402145 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.402023    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:07:52.402145 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.402032    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:07:52.404330 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.404318    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:07:52.404509 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.404495    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.404548 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.404531    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:07:52.405119 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.405096    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:07:52.405211 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.405125    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:07:52.405211 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.405134    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:07:52.405211 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.405148    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:07:52.405211 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.405160    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:07:52.405211 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.405170    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:07:52.407370 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.407355    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.407417 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.407381    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:07:52.408148 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.408128    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:07:52.408236 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.408154    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:07:52.408236 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.408181    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:07:52.425121 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.425034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.425121 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.425074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.425121 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.425101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32f26124355cba5ec51ff80a5481f572-config\") pod \"kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"32f26124355cba5ec51ff80a5481f572\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.430951 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.430930    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found" node="ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.433496 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.433471    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found" node="ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.491861 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.491828    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.526277 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.526239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.526436 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.526275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.526436 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.526322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32f26124355cba5ec51ff80a5481f572-config\") pod \"kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"32f26124355cba5ec51ff80a5481f572\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.526436 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.526342    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.526436 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.526359    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32f26124355cba5ec51ff80a5481f572-config\") pod \"kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"32f26124355cba5ec51ff80a5481f572\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.526436 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.526384    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.592886 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.592854    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.694249 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.693987    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.735604 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.735559    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.739026 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.739003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:52.796735 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.796687    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.897275 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.897237    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:52.941677 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:52.941646    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 21:07:52.942193 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.941826    2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab303f828c2034a3dad08796677c6e91-0df893518692fff8.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-86.us-west-2.compute.internal.18e45682f43b82a7\": read tcp 10.0.133.86:40714->3.132.102.228:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-86.us-west-2.compute.internal.18e45682f43b82a7  default   5045 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-86.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-86.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-86.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:07:52 +0000 UTC,LastTimestamp:2026-11-02 21:07:52.402027286 +0000 UTC m=+0.807052691,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-86.us-west-2.compute.internal,}"
Nov 02 21:07:52.998369 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:52.998299    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:53.098563 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:53.098529    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:53.191939 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.191906    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:07:53.193060 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.193044    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:07:53.193164 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.193076    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:07:53.193164 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.193088    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:07:53.199071 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:53.199048    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:53.273741 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.273698    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:02:52 +0000 UTC" deadline="2028-08-16 04:39:01.306466052 +0000 UTC"
Nov 02 21:07:53.273741 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.273733    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15655h31m8.032736531s"
Nov 02 21:07:53.299168 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:53.299145    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:53.326772 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.326745    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:07:53.400351 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:53.400313    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:53.501554 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:53.501511    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:53.596094 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.595954    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:07:53.602090 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:53.602062    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:53.696541 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.696509    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:07:53.702658 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:53.702637    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:07:53.723758 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.723732    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:07:53.821581 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.821536    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:53.884355 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.884285    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:07:53.885508 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.885490    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:07:53.947885 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.947865    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:07:53.955195 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:53.955130    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f26124355cba5ec51ff80a5481f572.slice/crio-6ec3711a66e817b4bce816161483d39b166c56a5145291e10214e65bfb56f885 WatchSource:0}: Error finding container 6ec3711a66e817b4bce816161483d39b166c56a5145291e10214e65bfb56f885: Status 404 returned error can't find the container with id 6ec3711a66e817b4bce816161483d39b166c56a5145291e10214e65bfb56f885
Nov 02 21:07:53.955562 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:53.955534    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65299bdb8e943305caf24f1bcda4b12f.slice/crio-83cddc74cea102b6aa974c9c5c094dee54bc7b01e929cdfaf219c2cf600343dc WatchSource:0}: Error finding container 83cddc74cea102b6aa974c9c5c094dee54bc7b01e929cdfaf219c2cf600343dc: Status 404 returned error can't find the container with id 83cddc74cea102b6aa974c9c5c094dee54bc7b01e929cdfaf219c2cf600343dc
Nov 02 21:07:53.962664 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.962639    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:07:53.993937 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:53.993902    2583 apiserver.go:52] "Watching apiserver"
Nov 02 21:07:54.020829 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.020791    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 21:07:54.100745 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.100715    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 21:07:54.101087 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.101068    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-rp6h8","openshift-multus/network-metrics-daemon-zf2t8","kube-system/global-pull-secret-syncer-zl2hk","kube-system/konnectivity-agent-qzbfv","openshift-cluster-node-tuning-operator/tuned-77cx8","openshift-network-diagnostics/network-check-target-plprl","openshift-ovn-kubernetes/ovnkube-node-2lfm8","openshift-multus/multus-cz9xl","openshift-image-registry/node-ca-sllxw","openshift-network-operator/iptables-alerter-phqtr","kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7","openshift-dns/node-resolver-zc285"]
Nov 02 21:07:54.105231 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.105204    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.107643 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.107620    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.110053 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.110035    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:54.110149 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.110108    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:07:54.112507 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.112484    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.114931 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.114913    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:07:54.118074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.118047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.120472 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.120456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.122783 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.122767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:07:54.122877 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.122840    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:07:54.125085 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.125069    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.127514 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.127497    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.129934 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.129916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.132425 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.132385    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"32f26124355cba5ec51ff80a5481f572","Type":"ContainerStarted","Data":"6ec3711a66e817b4bce816161483d39b166c56a5145291e10214e65bfb56f885"}
Nov 02 21:07:54.132504 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.132440    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"65299bdb8e943305caf24f1bcda4b12f","Type":"ContainerStarted","Data":"83cddc74cea102b6aa974c9c5c094dee54bc7b01e929cdfaf219c2cf600343dc"}
Nov 02 21:07:54.132504 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.132488    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.132579 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.132562    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:07:54.135347 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stph2\" (UniqueName: \"kubernetes.io/projected/8abc6131-3792-4669-bfe7-a1c88ee317a3-kube-api-access-stph2\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:54.135438 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135356    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-kubernetes\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.135438 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135385    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-registration-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.135438 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vm8\" (UniqueName: \"kubernetes.io/projected/a9941cc1-f486-4fbc-bc93-9b6070db00d3-kube-api-access-f2vm8\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.135598 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cnibin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.135598 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cni-binary-copy\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.135598 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-kubelet\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.135698 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.135698 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-var-lib-kubelet\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.135698 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-device-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.135786 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-k8s-cni-cncf-io\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.135786 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-hostroot\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.135863 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-conf-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.135863 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-host\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.135940 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-serviceca\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.135940 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.135940 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-socket-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.136067 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.135972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pjs7\" (UniqueName: \"kubernetes.io/projected/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kube-api-access-2pjs7\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.136067 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-netns\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.136067 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8444a0ef-d9fe-4605-97e3-6d6263228a11-agent-certs\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:07:54.136244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-sys\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-lib-modules\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.136244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-socket-dir-parent\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.136244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136241    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-multus\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.136494 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-daemon-config\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.136494 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-multus-certs\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.136494 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-tuned\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136494 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.136494 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-system-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.136494 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-etc-kubernetes\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.136494 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgmh\" (UniqueName: \"kubernetes.io/projected/1b38e492-9c4b-4190-8456-8a2b965ad3e7-kube-api-access-9vgmh\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.136494 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zs6t\" (UniqueName: \"kubernetes.io/projected/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-kube-api-access-6zs6t\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-modprobe-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysconfig\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-conf\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-systemd\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136628    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-run\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136700    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9941cc1-f486-4fbc-bc93-9b6070db00d3-hosts-file\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-host\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-tmp\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwwx\" (UniqueName: \"kubernetes.io/projected/60d81696-4204-4290-81f7-8cfe42e3bed0-kube-api-access-ddwwx\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-sys-fs\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9941cc1-f486-4fbc-bc93-9b6070db00d3-tmp-dir\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8444a0ef-d9fe-4605-97e3-6d6263228a11-konnectivity-ca\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:07:54.136916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.136924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-bin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.160398 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.160370    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 21:07:54.160524 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.160396    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 21:07:54.160524 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.160410    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 21:07:54.160594 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.160524    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8dkx\""
Nov 02 21:07:54.161494 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.161476    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 21:07:54.161580 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.161518    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 21:07:54.161580 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.161530    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 21:07:54.161670 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.161652    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbqzp\""
Nov 02 21:07:54.165810 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.165794    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 21:07:54.166002 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.165991    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 21:07:54.166189 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.166177    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9vsgv\""
Nov 02 21:07:54.168449 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.168434    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 21:07:54.168522 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.168476    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z9mmm\""
Nov 02 21:07:54.168719 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.168706    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 21:07:54.173964 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.173945    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:07:54.174047 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.174030    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 21:07:54.174097 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.174080    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fx9vz\""
Nov 02 21:07:54.174483 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.174469    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 21:07:54.174859 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.174842    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 21:07:54.174907 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.174843    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 21:07:54.174953 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.174909    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 21:07:54.174953 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.174928    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v96df\""
Nov 02 21:07:54.206653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.206616    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvdw\""
Nov 02 21:07:54.207411 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.207388    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 21:07:54.207514 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.207413    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 21:07:54.207514 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.207387    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 21:07:54.207514 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.207434    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 21:07:54.207514 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.207433    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9rn4m\""
Nov 02 21:07:54.207514 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.207485    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 21:07:54.207768 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.207741    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 21:07:54.211847 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.211833    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 21:07:54.212274 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.212251    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:07:54.212656 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.212637    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 21:07:54.212768 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.212665    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 21:07:54.212768 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.212690    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 21:07:54.212768 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.212692    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v679m"
Nov 02 21:07:54.212768 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.212709    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-894d2\""
Nov 02 21:07:54.212957 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.212905    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 21:07:54.222975 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.222954    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 21:07:54.238288 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.238398 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-dbus\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.238398 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-etc-kubernetes\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.238398 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-binary-copy\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.238398 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-socket-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.238398 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjs7\" (UniqueName: \"kubernetes.io/projected/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kube-api-access-2pjs7\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.238398 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238381    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-etc-kubernetes\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.238649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-daemon-config\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.238649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238452    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-script-lib\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.238649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-kubelet-config\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.238649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-system-cni-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.238649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238520    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.238649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238556    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-socket-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.238649 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-socket-dir-parent\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-bin\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238697    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-socket-dir-parent\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-env-overrides\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-tuned\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-system-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-systemd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-netd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238845    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-system-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scqv\" (UniqueName: \"kubernetes.io/projected/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-kube-api-access-2scqv\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zs6t\" (UniqueName: \"kubernetes.io/projected/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-kube-api-access-6zs6t\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.238965 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.238944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-systemd\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-run\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239070    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239095    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-systemd\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239073    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-daemon-config\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9941cc1-f486-4fbc-bc93-9b6070db00d3-tmp-dir\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239171    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-run\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-host-slash\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-host\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-sys-fs\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-cnibin\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239358    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-sys-fs\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9hr2\" (UniqueName: \"kubernetes.io/projected/12e057a9-159b-465f-9864-3d4521016ac6-kube-api-access-r9hr2\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.239415 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239382    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-host\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-log-socket\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239452    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-config\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239493    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9941cc1-f486-4fbc-bc93-9b6070db00d3-tmp-dir\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8444a0ef-d9fe-4605-97e3-6d6263228a11-konnectivity-ca\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-bin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-kubernetes\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-registration-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.239652    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239661    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-bin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239675    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-kubernetes\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vm8\" (UniqueName: \"kubernetes.io/projected/a9941cc1-f486-4fbc-bc93-9b6070db00d3-kube-api-access-f2vm8\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.239735    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:54.739706053 +0000 UTC m=+3.144731473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cnibin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cni-binary-copy\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.239870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239798    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-registration-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-conf-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239834    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cnibin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-kubelet\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239819    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-conf-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239885    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-var-lib-kubelet\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-device-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-k8s-cni-cncf-io\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239950    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-var-lib-kubelet\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.239977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240000    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-device-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-slash\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89fr2\" (UniqueName: \"kubernetes.io/projected/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-kube-api-access-89fr2\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240124    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-k8s-cni-cncf-io\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgmh\" (UniqueName: \"kubernetes.io/projected/1b38e492-9c4b-4190-8456-8a2b965ad3e7-kube-api-access-9vgmh\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-host\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240263    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-host\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.240627 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240267    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8444a0ef-d9fe-4605-97e3-6d6263228a11-konnectivity-ca\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-serviceca\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-netns\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-multus-certs\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240395    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cni-binary-copy\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240426    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-netns\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8444a0ef-d9fe-4605-97e3-6d6263228a11-agent-certs\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240503    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-multus-certs\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240528    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-sys\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-lib-modules\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-multus\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-netns\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240638    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-sys\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.241424 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-var-lib-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-node-log\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240745    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-multus\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240761    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-iptables-alerter-script\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-modprobe-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysconfig\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-conf\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240883    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240892    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-serviceca\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240907    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysconfig\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9941cc1-f486-4fbc-bc93-9b6070db00d3-hosts-file\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240960    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-modprobe-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.240972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-systemd-units\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-etc-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.242123 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovn-node-metrics-cert\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241068    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-conf\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241113    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9941cc1-f486-4fbc-bc93-9b6070db00d3-hosts-file\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-tmp\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241257    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-lib-modules\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwwx\" (UniqueName: \"kubernetes.io/projected/60d81696-4204-4290-81f7-8cfe42e3bed0-kube-api-access-ddwwx\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-ovn\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stph2\" (UniqueName: \"kubernetes.io/projected/8abc6131-3792-4669-bfe7-a1c88ee317a3-kube-api-access-stph2\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-kubelet\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-hostroot\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241683    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-kubelet\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241765    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-hostroot\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.242646 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.241840    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.243033 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.242834    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-tuned\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.243288 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.243271    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8444a0ef-d9fe-4605-97e3-6d6263228a11-agent-certs\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:07:54.243330 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.243286    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-tmp\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.297451 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.297413    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:07:54.297451 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.297447    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:07:54.297719 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.297466    2583 projected.go:196] Error preparing data for projected volume kube-api-access-t8lmb for pod openshift-network-diagnostics/network-check-target-plprl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:07:54.297719 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.297552    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb podName:71142023-8714-4106-8db9-e3522b32da9e nodeName:}" failed. No retries permitted until 2026-11-02 21:07:54.797529197 +0000 UTC m=+3.202554616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t8lmb" (UniqueName: "kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb") pod "network-check-target-plprl" (UID: "71142023-8714-4106-8db9-e3522b32da9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:07:54.299486 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.299321    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zs6t\" (UniqueName: \"kubernetes.io/projected/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-kube-api-access-6zs6t\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.299486 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.299339    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjs7\" (UniqueName: \"kubernetes.io/projected/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kube-api-access-2pjs7\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.299486 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.299430    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vm8\" (UniqueName: \"kubernetes.io/projected/a9941cc1-f486-4fbc-bc93-9b6070db00d3-kube-api-access-f2vm8\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.299745 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.299653    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgmh\" (UniqueName: \"kubernetes.io/projected/1b38e492-9c4b-4190-8456-8a2b965ad3e7-kube-api-access-9vgmh\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.299831 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.299809    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwwx\" (UniqueName: \"kubernetes.io/projected/60d81696-4204-4290-81f7-8cfe42e3bed0-kube-api-access-ddwwx\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.300259 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.300241    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stph2\" (UniqueName: \"kubernetes.io/projected/8abc6131-3792-4669-bfe7-a1c88ee317a3-kube-api-access-stph2\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:54.342085 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-kubelet\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342278 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.342278 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-slash\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342278 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342154    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-kubelet\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342278 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342160    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89fr2\" (UniqueName: \"kubernetes.io/projected/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-kube-api-access-89fr2\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.342278 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342278 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.342278 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-netns\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342278 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342239    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-slash\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-var-lib-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-node-log\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342328    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-netns\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-iptables-alerter-script\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342361    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-var-lib-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342265    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342369    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-node-log\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-systemd-units\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342456    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-systemd-units\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342634 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342562    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-etc-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.342916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342783    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.342916 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342819    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.342992 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.342964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-etc-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343035 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovn-node-metrics-cert\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343070 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-ovn\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343105 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.343150 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343121    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-ovn\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343150 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343153    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-dbus\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.343244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343202    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-iptables-alerter-script\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.343244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-binary-copy\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.343380 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.343243    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:07:54.343380 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343261    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-dbus\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.343380 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-script-lib\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343380 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.343320    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret podName:b9be1c6f-dae7-418e-ae7d-d0cd42578469 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:54.843282756 +0000 UTC m=+3.248308154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret") pod "global-pull-secret-syncer-zl2hk" (UID: "b9be1c6f-dae7-418e-ae7d-d0cd42578469") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:07:54.343380 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-kubelet-config\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.343380 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-system-cni-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-bin\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343452    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-kubelet-config\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-env-overrides\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343480    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-system-cni-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-systemd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-netd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343588    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-systemd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343546    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-bin\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343617    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-netd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.343653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scqv\" (UniqueName: \"kubernetes.io/projected/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-kube-api-access-2scqv\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-host-slash\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-cnibin\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hr2\" (UniqueName: \"kubernetes.io/projected/12e057a9-159b-465f-9864-3d4521016ac6-kube-api-access-r9hr2\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343735    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-host-slash\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-log-socket\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343770    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-cnibin\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-config\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343819    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-log-socket\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.344074 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.343878    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.344493 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.344467    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.344644 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.344607    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-binary-copy\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.344706 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.344689    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-script-lib\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.344800 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.344783    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-env-overrides\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.344870 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.344810    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-config\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.345639 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.345622    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovn-node-metrics-cert\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.404560 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.404482    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89fr2\" (UniqueName: \"kubernetes.io/projected/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-kube-api-access-89fr2\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.406313 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.406288    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scqv\" (UniqueName: \"kubernetes.io/projected/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-kube-api-access-2scqv\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.406454 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.406438    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hr2\" (UniqueName: \"kubernetes.io/projected/12e057a9-159b-465f-9864-3d4521016ac6-kube-api-access-r9hr2\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.417599 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.417577    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:07:54.428065 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:54.428037    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a48f66_64fa_40f0_a8ad_17ffcc05bd2f.slice/crio-36a600f3c16e2abc954257579b18aceec4eda40e85b5e8fcb038c0ed9f49c0f6 WatchSource:0}: Error finding container 36a600f3c16e2abc954257579b18aceec4eda40e85b5e8fcb038c0ed9f49c0f6: Status 404 returned error can't find the container with id 36a600f3c16e2abc954257579b18aceec4eda40e85b5e8fcb038c0ed9f49c0f6
Nov 02 21:07:54.428719 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.428679    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:07:54.433897 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.433880    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zc285"
Nov 02 21:07:54.436221 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.436204    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:07:54.437313 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:54.437284    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11c4c6b_5bf0_4203_b5c6_2287c0ff122c.slice/crio-4e2cd58226b177797b2793a400a896aa649159db0db4f54656bb6475f31e6911 WatchSource:0}: Error finding container 4e2cd58226b177797b2793a400a896aa649159db0db4f54656bb6475f31e6911: Status 404 returned error can't find the container with id 4e2cd58226b177797b2793a400a896aa649159db0db4f54656bb6475f31e6911
Nov 02 21:07:54.442464 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.442444    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:07:54.443681 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:54.443660    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9941cc1_f486_4fbc_bc93_9b6070db00d3.slice/crio-f5269b8e1bb3da3ee4d156f7758aa21e85a02dd3940b057c9b99aa59b14c48c0 WatchSource:0}: Error finding container f5269b8e1bb3da3ee4d156f7758aa21e85a02dd3940b057c9b99aa59b14c48c0: Status 404 returned error can't find the container with id f5269b8e1bb3da3ee4d156f7758aa21e85a02dd3940b057c9b99aa59b14c48c0
Nov 02 21:07:54.447482 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:54.447461    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8444a0ef_d9fe_4605_97e3_6d6263228a11.slice/crio-5890923407df8992f3d0e8dacf7800593f826b4adb22ff1df4933a7c75582188 WatchSource:0}: Error finding container 5890923407df8992f3d0e8dacf7800593f826b4adb22ff1df4933a7c75582188: Status 404 returned error can't find the container with id 5890923407df8992f3d0e8dacf7800593f826b4adb22ff1df4933a7c75582188
Nov 02 21:07:54.451884 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.451866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cz9xl"
Nov 02 21:07:54.452222 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:54.452203    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d81696_4204_4290_81f7_8cfe42e3bed0.slice/crio-1fe49b6a7f9dd1ca1f292a8763e605e30ca2da3be76202f32d73e43b1f80fb47 WatchSource:0}: Error finding container 1fe49b6a7f9dd1ca1f292a8763e605e30ca2da3be76202f32d73e43b1f80fb47: Status 404 returned error can't find the container with id 1fe49b6a7f9dd1ca1f292a8763e605e30ca2da3be76202f32d73e43b1f80fb47
Nov 02 21:07:54.460670 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:54.460648    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b38e492_9c4b_4190_8456_8a2b965ad3e7.slice/crio-b6002d2a8301d23f4894c58372942988e309f1ca0484e056bc56b68311903650 WatchSource:0}: Error finding container b6002d2a8301d23f4894c58372942988e309f1ca0484e056bc56b68311903650: Status 404 returned error can't find the container with id b6002d2a8301d23f4894c58372942988e309f1ca0484e056bc56b68311903650
Nov 02 21:07:54.473895 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.473866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:07:54.479019 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.478992    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:07:54.483381 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:54.483358    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211397c2_9e2b_48fe_9fb4_050a6cba9a5c.slice/crio-1cf460d885ddd8bff4394a61fbee03ac6b7d9f3f3fa6dd8ceaeaeaaa254b3df3 WatchSource:0}: Error finding container 1cf460d885ddd8bff4394a61fbee03ac6b7d9f3f3fa6dd8ceaeaeaaa254b3df3: Status 404 returned error can't find the container with id 1cf460d885ddd8bff4394a61fbee03ac6b7d9f3f3fa6dd8ceaeaeaaa254b3df3
Nov 02 21:07:54.488574 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:54.488551    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e057a9_159b_465f_9864_3d4521016ac6.slice/crio-6159573cd2493cc31b2978e50b973dd37d4a41c47a2699229ee62da99618d6e9 WatchSource:0}: Error finding container 6159573cd2493cc31b2978e50b973dd37d4a41c47a2699229ee62da99618d6e9: Status 404 returned error can't find the container with id 6159573cd2493cc31b2978e50b973dd37d4a41c47a2699229ee62da99618d6e9
Nov 02 21:07:54.490096 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.490080    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:07:54.499535 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:07:54.499513    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9118c60c_2783_4f6e_8f31_2f5bd6f8d684.slice/crio-f011932736ec69596fe0ab65960926916017f33d73a9613ba88fae4b53e21691 WatchSource:0}: Error finding container f011932736ec69596fe0ab65960926916017f33d73a9613ba88fae4b53e21691: Status 404 returned error can't find the container with id f011932736ec69596fe0ab65960926916017f33d73a9613ba88fae4b53e21691
Nov 02 21:07:54.747149 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.747030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:54.747300 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.747216    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:07:54.747300 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.747281    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:55.747267321 +0000 UTC m=+4.152292718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:07:54.849136 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.849099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:07:54.849318 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:54.849150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:54.849318 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.849245    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:07:54.849318 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.849277    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:07:54.849318 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.849296    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret podName:b9be1c6f-dae7-418e-ae7d-d0cd42578469 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:55.849282908 +0000 UTC m=+4.254308302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret") pod "global-pull-secret-syncer-zl2hk" (UID: "b9be1c6f-dae7-418e-ae7d-d0cd42578469") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:07:54.849318 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.849301    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:07:54.849318 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.849314    2583 projected.go:196] Error preparing data for projected volume kube-api-access-t8lmb for pod openshift-network-diagnostics/network-check-target-plprl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:07:54.849530 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:54.849361    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb podName:71142023-8714-4106-8db9-e3522b32da9e nodeName:}" failed. No retries permitted until 2026-11-02 21:07:55.849348912 +0000 UTC m=+4.254374309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8lmb" (UniqueName: "kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb") pod "network-check-target-plprl" (UID: "71142023-8714-4106-8db9-e3522b32da9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:07:55.109771 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.109727    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp6h8" event={"ID":"12e057a9-159b-465f-9864-3d4521016ac6","Type":"ContainerStarted","Data":"6159573cd2493cc31b2978e50b973dd37d4a41c47a2699229ee62da99618d6e9"}
Nov 02 21:07:55.112399 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.112364    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-77cx8" event={"ID":"60d81696-4204-4290-81f7-8cfe42e3bed0","Type":"ContainerStarted","Data":"1fe49b6a7f9dd1ca1f292a8763e605e30ca2da3be76202f32d73e43b1f80fb47"}
Nov 02 21:07:55.113973 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.113936    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzbfv" event={"ID":"8444a0ef-d9fe-4605-97e3-6d6263228a11","Type":"ContainerStarted","Data":"5890923407df8992f3d0e8dacf7800593f826b4adb22ff1df4933a7c75582188"}
Nov 02 21:07:55.115284 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.115249    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zc285" event={"ID":"a9941cc1-f486-4fbc-bc93-9b6070db00d3","Type":"ContainerStarted","Data":"f5269b8e1bb3da3ee4d156f7758aa21e85a02dd3940b057c9b99aa59b14c48c0"}
Nov 02 21:07:55.116612 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.116576    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" event={"ID":"9118c60c-2783-4f6e-8f31-2f5bd6f8d684","Type":"ContainerStarted","Data":"f011932736ec69596fe0ab65960926916017f33d73a9613ba88fae4b53e21691"}
Nov 02 21:07:55.117945 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.117921    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-phqtr" event={"ID":"211397c2-9e2b-48fe-9fb4-050a6cba9a5c","Type":"ContainerStarted","Data":"1cf460d885ddd8bff4394a61fbee03ac6b7d9f3f3fa6dd8ceaeaeaaa254b3df3"}
Nov 02 21:07:55.119479 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.119426    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cz9xl" event={"ID":"1b38e492-9c4b-4190-8456-8a2b965ad3e7","Type":"ContainerStarted","Data":"b6002d2a8301d23f4894c58372942988e309f1ca0484e056bc56b68311903650"}
Nov 02 21:07:55.121099 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.121068    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7" event={"ID":"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c","Type":"ContainerStarted","Data":"4e2cd58226b177797b2793a400a896aa649159db0db4f54656bb6475f31e6911"}
Nov 02 21:07:55.122542 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.122515    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sllxw" event={"ID":"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f","Type":"ContainerStarted","Data":"36a600f3c16e2abc954257579b18aceec4eda40e85b5e8fcb038c0ed9f49c0f6"}
Nov 02 21:07:55.216422 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.216370    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:02:54 +0000 UTC" deadline="2028-08-06 18:57:56.479500736 +0000 UTC"
Nov 02 21:07:55.216422 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.216420    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15429h50m1.263085856s"
Nov 02 21:07:55.760769 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.760620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:55.760959 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:55.760811    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:07:55.760959 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:55.760873    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:57.760854341 +0000 UTC m=+6.165879739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:07:55.861622 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.861581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:07:55.861840 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:55.861637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:55.861840 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:55.861794    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:07:55.861963 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:55.861862    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret podName:b9be1c6f-dae7-418e-ae7d-d0cd42578469 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:57.861841886 +0000 UTC m=+6.266867287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret") pod "global-pull-secret-syncer-zl2hk" (UID: "b9be1c6f-dae7-418e-ae7d-d0cd42578469") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:07:55.861963 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:55.861882    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:07:55.861963 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:55.861907    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:07:55.861963 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:55.861923    2583 projected.go:196] Error preparing data for projected volume kube-api-access-t8lmb for pod openshift-network-diagnostics/network-check-target-plprl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:07:55.862213 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:55.861989    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb podName:71142023-8714-4106-8db9-e3522b32da9e nodeName:}" failed. No retries permitted until 2026-11-02 21:07:57.861969074 +0000 UTC m=+6.266994485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8lmb" (UniqueName: "kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb") pod "network-check-target-plprl" (UID: "71142023-8714-4106-8db9-e3522b32da9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:07:56.104601 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:56.103515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:56.104601 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:56.103661    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:07:56.104601 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:56.103753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:07:56.104601 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:56.103832    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:07:56.104601 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:56.103904    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:56.104601 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:56.103981    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:07:56.217420 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:56.217338    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:02:54 +0000 UTC" deadline="2028-07-29 10:25:12.927294252 +0000 UTC"
Nov 02 21:07:56.217420 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:56.217408    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15229h17m16.709890538s"
Nov 02 21:07:57.780255 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:57.780216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:57.780972 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:57.780366    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:07:57.780972 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:57.780435    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:01.780415653 +0000 UTC m=+10.185441051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:07:57.881452 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:57.881373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:07:57.881452 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:57.881430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:57.881702 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:57.881681    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:07:57.882442 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:57.881759    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret podName:b9be1c6f-dae7-418e-ae7d-d0cd42578469 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:01.881739748 +0000 UTC m=+10.286765148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret") pod "global-pull-secret-syncer-zl2hk" (UID: "b9be1c6f-dae7-418e-ae7d-d0cd42578469") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:07:57.882442 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:57.882281    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:07:57.882442 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:57.882307    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:07:57.882442 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:57.882320    2583 projected.go:196] Error preparing data for projected volume kube-api-access-t8lmb for pod openshift-network-diagnostics/network-check-target-plprl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:07:57.882442 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:57.882386    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb podName:71142023-8714-4106-8db9-e3522b32da9e nodeName:}" failed. No retries permitted until 2026-11-02 21:08:01.882369618 +0000 UTC m=+10.287395015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8lmb" (UniqueName: "kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb") pod "network-check-target-plprl" (UID: "71142023-8714-4106-8db9-e3522b32da9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:07:58.103278 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:58.103210    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:07:58.103487 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:58.103369    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:07:58.103757 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:58.103741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:07:58.103842 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:58.103826    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:07:58.103898 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:58.103868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:07:58.103950 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:07:58.103914    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:07:59.139418 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:59.139131    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"32f26124355cba5ec51ff80a5481f572","Type":"ContainerStarted","Data":"c49e25f1faf0dc4e264eb38472058a43b169bbae5ac936e9ce9c96f95cf8a663"}
Nov 02 21:07:59.143512 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:59.143442    2583 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="387b3467cba5d40b40a104f3e1eb338063c8db50adfebb57da2e391952433c70" exitCode=0
Nov 02 21:07:59.143673 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:59.143551    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp6h8" event={"ID":"12e057a9-159b-465f-9864-3d4521016ac6","Type":"ContainerDied","Data":"387b3467cba5d40b40a104f3e1eb338063c8db50adfebb57da2e391952433c70"}
Nov 02 21:07:59.149306 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:59.149270    2583 generic.go:356] "Generic (PLEG): container finished" podID="65299bdb8e943305caf24f1bcda4b12f" containerID="2f14b63c86de9e3dc11179c211a02ab1232a49ccb04206e55be1db614dfdc606" exitCode=0
Nov 02 21:07:59.149435 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:59.149326    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"65299bdb8e943305caf24f1bcda4b12f","Type":"ContainerDied","Data":"2f14b63c86de9e3dc11179c211a02ab1232a49ccb04206e55be1db614dfdc606"}
Nov 02 21:07:59.257613 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:07:59.257544    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal" podStartSLOduration=5.25752376 podStartE2EDuration="5.25752376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:59.256568526 +0000 UTC m=+7.661593946" watchObservedRunningTime="2026-11-02 21:07:59.25752376 +0000 UTC m=+7.662549180"
Nov 02 21:08:00.102152 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:00.101503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:00.102152 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:00.101647    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:00.102152 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:00.102033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:00.102813 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:00.102573    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:00.102813 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:00.102670    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:00.103064 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:00.103022    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:00.156579 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:00.156546    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzbfv" event={"ID":"8444a0ef-d9fe-4605-97e3-6d6263228a11","Type":"ContainerStarted","Data":"84937c046ed3a5038473c7320b1d8c0e398fda8b3656f12896b1dba8f31ced52"}
Nov 02 21:08:00.162597 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:00.162544    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"65299bdb8e943305caf24f1bcda4b12f","Type":"ContainerStarted","Data":"ff613c5444f1fccade6a761281c12749398a423aa11e5aa5835ccec2590c0ac9"}
Nov 02 21:08:00.267508 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:00.267459    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qzbfv" podStartSLOduration=3.031790499 podStartE2EDuration="8.267441801s" totalImagesPullingTime="5.235651302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:00.266841665 +0000 UTC m=+8.671867079" watchObservedRunningTime="2026-11-02 21:08:00.267441801 +0000 UTC m=+8.672467214"
Nov 02 21:08:01.170693 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:01.170656    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sllxw" event={"ID":"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f","Type":"ContainerStarted","Data":"561a3362e6f3107aa97514f8d0296bcffbe39451f171147a7dd1bfa9494187a6"}
Nov 02 21:08:01.278673 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:01.278623    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal" podStartSLOduration=7.278605898 podStartE2EDuration="7.278605898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:00.375817976 +0000 UTC m=+8.780843392" watchObservedRunningTime="2026-11-02 21:08:01.278605898 +0000 UTC m=+9.683631312"
Nov 02 21:08:01.816168 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:01.816092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:01.816336 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:01.816269    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:08:01.816336 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:01.816330    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:09.816313271 +0000 UTC m=+18.221338667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:08:01.916898 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:01.916853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:01.917077 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:01.916935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:01.917155 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:01.917130    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:08:01.917237 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:01.917194    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret podName:b9be1c6f-dae7-418e-ae7d-d0cd42578469 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:09.917176026 +0000 UTC m=+18.322201433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret") pod "global-pull-secret-syncer-zl2hk" (UID: "b9be1c6f-dae7-418e-ae7d-d0cd42578469") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:08:01.917639 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:01.917621    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:08:01.917740 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:01.917647    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:08:01.917740 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:01.917659    2583 projected.go:196] Error preparing data for projected volume kube-api-access-t8lmb for pod openshift-network-diagnostics/network-check-target-plprl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:08:01.917740 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:01.917704    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb podName:71142023-8714-4106-8db9-e3522b32da9e nodeName:}" failed. No retries permitted until 2026-11-02 21:08:09.91768999 +0000 UTC m=+18.322715407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8lmb" (UniqueName: "kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb") pod "network-check-target-plprl" (UID: "71142023-8714-4106-8db9-e3522b32da9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:08:02.105200 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:02.103582    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:02.105200 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:02.103710    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:02.105200 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:02.104083    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:02.105200 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:02.104182    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:02.105200 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:02.104239    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:02.105200 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:02.104306    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:02.175364 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:02.175329    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7" event={"ID":"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c","Type":"ContainerStarted","Data":"ee1b3b67ef047715c69cd705ceaaab6b9de0129fc7f88d90968023f4f17297e8"}
Nov 02 21:08:03.337452 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:03.337375    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 21:08:04.100612 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.100532    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:04.100612 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.100571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:04.100816 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:04.100696    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:04.100816 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.100759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:04.100816 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:04.100760    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:04.100971 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:04.100836    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:04.185005 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.184969    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zc285" event={"ID":"a9941cc1-f486-4fbc-bc93-9b6070db00d3","Type":"ContainerStarted","Data":"e97f207ea891f4b5f3a1e46d185ca85a788b98be140bf9838f2dd6e16dad5dad"}
Nov 02 21:08:04.188701 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.188645    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cz9xl" event={"ID":"1b38e492-9c4b-4190-8456-8a2b965ad3e7","Type":"ContainerStarted","Data":"bda5cdcfb08a4a2d9ab3c50fa132f2bed586a6ec352bc73fb44c9bc490a48528"}
Nov 02 21:08:04.192467 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.192435    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7" event={"ID":"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c","Type":"ContainerStarted","Data":"fb7bb44ec67f9054a151b98f8deba55f9781519297e1f24db0f0b8a7a00e6715"}
Nov 02 21:08:04.195243 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.195215    2583 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="fd73cd889359fda6ae3d8ee007ecb247cda4226755e6140d1bf011faa004e500" exitCode=0
Nov 02 21:08:04.195345 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.195251    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp6h8" event={"ID":"12e057a9-159b-465f-9864-3d4521016ac6","Type":"ContainerDied","Data":"fd73cd889359fda6ae3d8ee007ecb247cda4226755e6140d1bf011faa004e500"}
Nov 02 21:08:04.234137 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.233983    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:08:03.337393941Z","UUID":"5f973c06-c2ce-4702-8e6d-73a60bc75e6a","Handler":null,"Name":"","Endpoint":""}
Nov 02 21:08:04.236788 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.236761    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 21:08:04.236788 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.236798    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 21:08:04.310407 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.310359    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sllxw" podStartSLOduration=6.151771514 podStartE2EDuration="12.310345304s" totalImagesPullingTime="6.15857379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:01.282070064 +0000 UTC m=+9.687095481" watchObservedRunningTime="2026-11-02 21:08:04.310345304 +0000 UTC m=+12.715370717"
Nov 02 21:08:04.310563 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.310533    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zc285" podStartSLOduration=3.696078361 podStartE2EDuration="12.310529042s" totalImagesPullingTime="8.614450681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:04.309870871 +0000 UTC m=+12.714896285" watchObservedRunningTime="2026-11-02 21:08:04.310529042 +0000 UTC m=+12.715554454"
Nov 02 21:08:04.420216 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.420168    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cz9xl" podStartSLOduration=3.347461535 podStartE2EDuration="12.420148918s" totalImagesPullingTime="9.072687383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:04.419394311 +0000 UTC m=+12.824419725" watchObservedRunningTime="2026-11-02 21:08:04.420148918 +0000 UTC m=+12.825174333"
Nov 02 21:08:04.962446 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.962397    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:08:04.963192 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:04.963170    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:08:05.199949 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:05.199868    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-phqtr" event={"ID":"211397c2-9e2b-48fe-9fb4-050a6cba9a5c","Type":"ContainerStarted","Data":"b8e7f3f081934aad13096e7107f7026d420939e0ccfcc331fca0d8c6d4ecd321"}
Nov 02 21:08:05.203586 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:05.203556    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7" event={"ID":"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c","Type":"ContainerStarted","Data":"3fda6c940cc9354028048ff1ad69fd65272de05a85d38e5ebfb3abd127a55d64"}
Nov 02 21:08:05.204202 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:05.204154    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:08:05.204398 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:05.204382    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:08:05.309571 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:05.309521    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-phqtr" podStartSLOduration=4.156607819 podStartE2EDuration="13.309505786s" totalImagesPullingTime="9.152897967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:05.309476936 +0000 UTC m=+13.714502351" watchObservedRunningTime="2026-11-02 21:08:05.309505786 +0000 UTC m=+13.714531198"
Nov 02 21:08:05.534619 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:05.534421    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7" podStartSLOduration=3.418767766 podStartE2EDuration="13.534406846s" totalImagesPullingTime="10.11563908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:05.533949716 +0000 UTC m=+13.938975132" watchObservedRunningTime="2026-11-02 21:08:05.534406846 +0000 UTC m=+13.939432259"
Nov 02 21:08:06.100607 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:06.100578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:06.100771 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:06.100693    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:06.100771 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:06.100767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:06.100853 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:06.100825    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:06.100986 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:06.100971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:06.101068 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:06.101044    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:06.208104 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:06.208026    2583 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="c018a0a17a4ff11b78e1c949647347e0cd54e43cf66ae6c22884a93ca6a0eb1c" exitCode=0
Nov 02 21:08:06.208290 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:06.208095    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp6h8" event={"ID":"12e057a9-159b-465f-9864-3d4521016ac6","Type":"ContainerDied","Data":"c018a0a17a4ff11b78e1c949647347e0cd54e43cf66ae6c22884a93ca6a0eb1c"}
Nov 02 21:08:06.209808 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:06.209782    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-77cx8" event={"ID":"60d81696-4204-4290-81f7-8cfe42e3bed0","Type":"ContainerStarted","Data":"c9d6e295dd2e547cb4385a02922d248b707d655534c1b8f50fcaff26bc6b0c13"}
Nov 02 21:08:06.438518 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:06.438458    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-77cx8" podStartSLOduration=3.036850331 podStartE2EDuration="14.438439334s" totalImagesPullingTime="11.401589003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:06.437781723 +0000 UTC m=+14.842807137" watchObservedRunningTime="2026-11-02 21:08:06.438439334 +0000 UTC m=+14.843464750"
Nov 02 21:08:08.101211 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:08.101049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:08.101211 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:08.101206    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:08.102217 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:08.101391    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:08.102217 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:08.101449    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:08.102217 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:08.101710    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:08.104288 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:08.102875    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:08.219393 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:08.219356    2583 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="8050bd793e1694f6586d68c913699c42ff7639e31dd9aef359ca234f2833a32e" exitCode=0
Nov 02 21:08:08.219546 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:08.219419    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp6h8" event={"ID":"12e057a9-159b-465f-9864-3d4521016ac6","Type":"ContainerDied","Data":"8050bd793e1694f6586d68c913699c42ff7639e31dd9aef359ca234f2833a32e"}
Nov 02 21:08:09.882621 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:09.882392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:09.883126 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:09.882537    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:08:09.883126 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:09.882747    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:25.882725385 +0000 UTC m=+34.287750779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 21:08:09.983382 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:09.983347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:09.983553 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:09.983393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:09.983553 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:09.983495    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 21:08:09.983553 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:09.983527    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 21:08:09.983553 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:09.983550    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 21:08:09.983732 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:09.983561    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret podName:b9be1c6f-dae7-418e-ae7d-d0cd42578469 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:25.983541246 +0000 UTC m=+34.388566650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret") pod "global-pull-secret-syncer-zl2hk" (UID: "b9be1c6f-dae7-418e-ae7d-d0cd42578469") : object "kube-system"/"original-pull-secret" not registered
Nov 02 21:08:09.983732 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:09.983563    2583 projected.go:196] Error preparing data for projected volume kube-api-access-t8lmb for pod openshift-network-diagnostics/network-check-target-plprl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:08:09.983732 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:09.983613    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb podName:71142023-8714-4106-8db9-e3522b32da9e nodeName:}" failed. No retries permitted until 2026-11-02 21:08:25.983598025 +0000 UTC m=+34.388623437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8lmb" (UniqueName: "kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb") pod "network-check-target-plprl" (UID: "71142023-8714-4106-8db9-e3522b32da9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 21:08:10.100289 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:10.100257    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:10.100289 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:10.100257    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:10.100465 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:10.100413    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:10.100465 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:10.100459    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:10.100543 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:10.100531    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:10.100607 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:10.100591    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:11.228662 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:11.228472    2583 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="1b95a9893290c6ca5f91a8a6c17d2f44fff9d11bc0f67619b96a9787b0870edc" exitCode=0
Nov 02 21:08:11.229228 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:11.228554    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp6h8" event={"ID":"12e057a9-159b-465f-9864-3d4521016ac6","Type":"ContainerDied","Data":"1b95a9893290c6ca5f91a8a6c17d2f44fff9d11bc0f67619b96a9787b0870edc"}
Nov 02 21:08:11.231705 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:11.231684    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" event={"ID":"9118c60c-2783-4f6e-8f31-2f5bd6f8d684","Type":"ContainerStarted","Data":"b4e670b98ce0e3c57388e3d3487630dbdf89f2e7bccf7a3cc51fa8625a803c79"}
Nov 02 21:08:11.231790 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:11.231756    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" event={"ID":"9118c60c-2783-4f6e-8f31-2f5bd6f8d684","Type":"ContainerStarted","Data":"cc7bfd639884ec226fc44bc7f6ffa0be972266b1bd12323516c8b5cfab6a010e"}
Nov 02 21:08:11.231842 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:11.231804    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" event={"ID":"9118c60c-2783-4f6e-8f31-2f5bd6f8d684","Type":"ContainerStarted","Data":"25fb0aea315eec2dda5f8b11f6e19ad027f71236da68b5fc4f25164324f32369"}
Nov 02 21:08:11.231894 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:11.231853    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" event={"ID":"9118c60c-2783-4f6e-8f31-2f5bd6f8d684","Type":"ContainerStarted","Data":"8802370e907250669fe8973224cb47f641099d2648c1c0389584c3bcc31c56e4"}
Nov 02 21:08:11.231983 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:11.231900    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" event={"ID":"9118c60c-2783-4f6e-8f31-2f5bd6f8d684","Type":"ContainerStarted","Data":"0451ae51a1c73bcead1e4b7ce56ef49c6514c1efbf8e2cfe1b73980e217fd528"}
Nov 02 21:08:11.231983 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:11.231945    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" event={"ID":"9118c60c-2783-4f6e-8f31-2f5bd6f8d684","Type":"ContainerStarted","Data":"534fb7dfbc5ca5088012b3309013db8eb0ce1f63681b787801bac6564de27c59"}
Nov 02 21:08:12.101052 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:12.101022    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:12.101233 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:12.101154    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:12.101233 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:12.101163    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:12.101304 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:12.101242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:12.101304 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:12.101242    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:12.101357 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:12.101341    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:12.236030 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:12.235995    2583 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="23abb1e9877c10ffc7f7858e4c4cd60b4c97462e75feba500f408c4fa9e3c0d7" exitCode=0
Nov 02 21:08:12.236486 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:12.236038    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp6h8" event={"ID":"12e057a9-159b-465f-9864-3d4521016ac6","Type":"ContainerDied","Data":"23abb1e9877c10ffc7f7858e4c4cd60b4c97462e75feba500f408c4fa9e3c0d7"}
Nov 02 21:08:13.241727 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:13.241690    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rp6h8" event={"ID":"12e057a9-159b-465f-9864-3d4521016ac6","Type":"ContainerStarted","Data":"155bfc671f0604b064902d4e012b0ba32637feb4d4ba50a8eb81c41480daf60f"}
Nov 02 21:08:13.244738 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:13.244713    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" event={"ID":"9118c60c-2783-4f6e-8f31-2f5bd6f8d684","Type":"ContainerStarted","Data":"8e69fb861f05775003ba4094a73dfbf2ab5894b0e2ecdee6352c6cad4e0a0c3d"}
Nov 02 21:08:13.354498 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:13.354447    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rp6h8" podStartSLOduration=8.094013107 podStartE2EDuration="21.354434014s" totalImagesPullingTime="13.260420907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:13.354386963 +0000 UTC m=+21.759412377" watchObservedRunningTime="2026-11-02 21:08:13.354434014 +0000 UTC m=+21.759459427"
Nov 02 21:08:14.100258 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:14.100220    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:14.100258 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:14.100244    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:14.100441 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:14.100339    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:14.100441 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:14.100422    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:14.100524 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:14.100479    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:14.100552 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:14.100536    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:15.252189 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:15.252156    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" event={"ID":"9118c60c-2783-4f6e-8f31-2f5bd6f8d684","Type":"ContainerStarted","Data":"fc3b951f7f3445633d0d5d3e0fbb057c2d1e294ea62f73faf1aa799a86a59a49"}
Nov 02 21:08:15.252887 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:15.252413    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:15.267509 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:15.267491    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:15.512393 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:15.512331    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8" podStartSLOduration=7.822131739 podStartE2EDuration="23.512312706s" totalImagesPullingTime="15.690180967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:15.463431381 +0000 UTC m=+23.868456795" watchObservedRunningTime="2026-11-02 21:08:15.512312706 +0000 UTC m=+23.917338121"
Nov 02 21:08:16.100767 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:16.100722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:16.101071 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:16.100722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:16.101071 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:16.100836    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:16.101071 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:16.100864    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:16.101071 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:16.100971    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:16.101071 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:16.101030    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:16.257529 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:16.257499    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:16.257915 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:16.257693    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:16.272331 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:16.272302    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:17.382961 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:17.382754    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-plprl"]
Nov 02 21:08:17.383425 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:17.383050    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:17.383425 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:17.383176    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:17.383425 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:17.383396    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zl2hk"]
Nov 02 21:08:17.383604 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:17.383489    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:17.383604 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:17.383586    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:17.384759 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:17.384740    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zf2t8"]
Nov 02 21:08:17.384852 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:17.384811    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:17.384907 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:17.384887    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:19.100149 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:19.100101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:19.100493 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:19.100183    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:19.100493 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:19.100215    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:19.100493 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:19.100309    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:19.100493 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:19.100333    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:19.100493 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:19.100418    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:21.100600 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:21.100551    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:21.101044 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:21.100616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:21.101044 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:21.100551    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:21.101044 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:21.100679    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:21.101044 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:21.100736    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:21.101044 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:21.100785    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:23.100364 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.100335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:23.100364 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.100356    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:23.100746 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.100366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:23.100746 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:23.100453    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-plprl" podUID="71142023-8714-4106-8db9-e3522b32da9e"
Nov 02 21:08:23.100746 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:23.100507    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zl2hk" podUID="b9be1c6f-dae7-418e-ae7d-d0cd42578469"
Nov 02 21:08:23.100746 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:23.100615    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf2t8" podUID="8abc6131-3792-4669-bfe7-a1c88ee317a3"
Nov 02 21:08:23.405638 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.405565    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeReady"
Nov 02 21:08:23.405773 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.405717    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 21:08:23.495887 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.495848    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n64lc"]
Nov 02 21:08:23.499008 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.498988    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:23.513461 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.513440    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wf86r"]
Nov 02 21:08:23.516290 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.516277    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ctfx4"]
Nov 02 21:08:23.516410 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.516394    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.519400 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.519384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.542786 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.542759    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8924h-85j4m"]
Nov 02 21:08:23.545545 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.545527    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.549471 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.549456    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 21:08:23.549569 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.549487    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 21:08:23.549737 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.549724    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d9psf\""
Nov 02 21:08:23.552338 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.552320    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 21:08:23.553389 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.553369    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n64lc"]
Nov 02 21:08:23.579950 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.579924    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wf86r"]
Nov 02 21:08:23.583844 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.583818    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ctfx4"]
Nov 02 21:08:23.641065 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.641033    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8924h-85j4m"]
Nov 02 21:08:23.649580 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.649540    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 21:08:23.649580 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.649567    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 21:08:23.649580 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.649589    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 21:08:23.649795 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.649605    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 21:08:23.649795 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.649605    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 21:08:23.650203 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.650183    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 21:08:23.655506 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.655493    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 21:08:23.678429 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678328    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.678429 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njx2v\" (UniqueName: \"kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.678429 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/178af0c6-6aaf-4a57-9ed4-25988a9bac33-config-volume\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.678602 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzls\" (UniqueName: \"kubernetes.io/projected/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-api-access-xwzls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.678602 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678538    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-crio-socket\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.678602 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.678602 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-data-volume\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.678602 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.678836 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178af0c6-6aaf-4a57-9ed4-25988a9bac33-tmp-dir\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.678836 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678772    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6h5\" (UniqueName: \"kubernetes.io/projected/178af0c6-6aaf-4a57-9ed4-25988a9bac33-kube-api-access-jn6h5\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.678836 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678801    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.678836 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-cert\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:23.678970 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgq4\" (UniqueName: \"kubernetes.io/projected/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-kube-api-access-9tgq4\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:23.678970 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.678970 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.678884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.688151 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.688130    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5jdjp\""
Nov 02 21:08:23.688151 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.688136    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9hv44\""
Nov 02 21:08:23.697638 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.697619    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-58fxm\""
Nov 02 21:08:23.699133 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.699097    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 21:08:23.779839 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.779807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6h5\" (UniqueName: \"kubernetes.io/projected/178af0c6-6aaf-4a57-9ed4-25988a9bac33-kube-api-access-jn6h5\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.780012 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.779846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.780012 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.779868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-cert\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:23.780012 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.779943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgq4\" (UniqueName: \"kubernetes.io/projected/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-kube-api-access-9tgq4\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:23.780012 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.779981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.780012 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.780302 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:23.780136    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:23.780302 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:23.780169    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:08:23.780302 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:23.780208    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.280188516 +0000 UTC m=+32.685213922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:23.780302 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:23.780235    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls podName:178af0c6-6aaf-4a57-9ed4-25988a9bac33 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.280221811 +0000 UTC m=+32.685247204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls") pod "dns-default-wf86r" (UID: "178af0c6-6aaf-4a57-9ed4-25988a9bac33") : secret "dns-default-metrics-tls" not found
Nov 02 21:08:23.780302 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780254    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.780302 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njx2v\" (UniqueName: \"kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.780302 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/178af0c6-6aaf-4a57-9ed4-25988a9bac33-config-volume\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780311    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzls\" (UniqueName: \"kubernetes.io/projected/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-api-access-xwzls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-crio-socket\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-data-volume\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178af0c6-6aaf-4a57-9ed4-25988a9bac33-tmp-dir\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:23.780530    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780562    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-crio-socket\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.780630 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:23.780598    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.280584157 +0000 UTC m=+32.685609568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : configmap "kube-rbac-proxy" not found
Nov 02 21:08:23.781179 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780812    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178af0c6-6aaf-4a57-9ed4-25988a9bac33-tmp-dir\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.781179 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.780847    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-data-volume\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.781179 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.781010    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/178af0c6-6aaf-4a57-9ed4-25988a9bac33-config-volume\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.781179 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.781169    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.781179 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.781173    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.783842 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.783822    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-cert\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:23.837389 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.837359    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzls\" (UniqueName: \"kubernetes.io/projected/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-api-access-xwzls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:23.837520 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.837371    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6h5\" (UniqueName: \"kubernetes.io/projected/178af0c6-6aaf-4a57-9ed4-25988a9bac33-kube-api-access-jn6h5\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:23.837520 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.837452    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgq4\" (UniqueName: \"kubernetes.io/projected/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-kube-api-access-9tgq4\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:23.837685 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.837664    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njx2v\" (UniqueName: \"kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:23.857327 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:23.857299    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:24.110622 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.110593    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:24.236449 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.236418    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8924h-85j4m"]
Nov 02 21:08:24.239581 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:08:24.239554    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283a86d8_9e77_4f36_8fa9_0bd380ec043f.slice/crio-e3e9926d4be3dcd486d78f7b6a6d146e486b71c4bf81a33f1349044e895298f4 WatchSource:0}: Error finding container e3e9926d4be3dcd486d78f7b6a6d146e486b71c4bf81a33f1349044e895298f4: Status 404 returned error can't find the container with id e3e9926d4be3dcd486d78f7b6a6d146e486b71c4bf81a33f1349044e895298f4
Nov 02 21:08:24.275159 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.275128    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8924h-85j4m" event={"ID":"283a86d8-9e77-4f36-8fa9-0bd380ec043f","Type":"ContainerStarted","Data":"e3e9926d4be3dcd486d78f7b6a6d146e486b71c4bf81a33f1349044e895298f4"}
Nov 02 21:08:24.284421 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.284393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:24.284591 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.284473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:24.284591 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.284500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:24.284591 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:24.284511    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 21:08:24.284591 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:24.284564    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:25.2845461 +0000 UTC m=+33.689571524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : configmap "kube-rbac-proxy" not found
Nov 02 21:08:24.284759 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:24.284614    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:24.284759 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:24.284613    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:08:24.284759 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:24.284663    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:25.284647882 +0000 UTC m=+33.689673289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:24.284759 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:24.284684    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls podName:178af0c6-6aaf-4a57-9ed4-25988a9bac33 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:25.284674437 +0000 UTC m=+33.689699830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls") pod "dns-default-wf86r" (UID: "178af0c6-6aaf-4a57-9ed4-25988a9bac33") : secret "dns-default-metrics-tls" not found
Nov 02 21:08:24.508968 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.508877    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n64lc"]
Nov 02 21:08:24.511671 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:08:24.511640    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8cdafe_c2f7_433c_aac4_f4c8c6adf870.slice/crio-e8f3a26dfe0f75d02835d2804613f920517da798932f531f2a029cdf4a4f4a26 WatchSource:0}: Error finding container e8f3a26dfe0f75d02835d2804613f920517da798932f531f2a029cdf4a4f4a26: Status 404 returned error can't find the container with id e8f3a26dfe0f75d02835d2804613f920517da798932f531f2a029cdf4a4f4a26
Nov 02 21:08:24.765416 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.765381    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"]
Nov 02 21:08:24.797393 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.797366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"
Nov 02 21:08:24.820319 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.820289    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"]
Nov 02 21:08:24.847653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.847629    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 21:08:24.847653 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.847630    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 21:08:24.847839 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.847722    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jb7zz\""
Nov 02 21:08:24.890377 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.890348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6qmr\" (UniqueName: \"kubernetes.io/projected/7740f0f0-41ba-4325-9a49-1ff07a8c8475-kube-api-access-m6qmr\") pod \"migrator-66df5d496b-lx92f\" (UID: \"7740f0f0-41ba-4325-9a49-1ff07a8c8475\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"
Nov 02 21:08:24.990696 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:24.990661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6qmr\" (UniqueName: \"kubernetes.io/projected/7740f0f0-41ba-4325-9a49-1ff07a8c8475-kube-api-access-m6qmr\") pod \"migrator-66df5d496b-lx92f\" (UID: \"7740f0f0-41ba-4325-9a49-1ff07a8c8475\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"
Nov 02 21:08:25.045500 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.045430    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6qmr\" (UniqueName: \"kubernetes.io/projected/7740f0f0-41ba-4325-9a49-1ff07a8c8475-kube-api-access-m6qmr\") pod \"migrator-66df5d496b-lx92f\" (UID: \"7740f0f0-41ba-4325-9a49-1ff07a8c8475\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"
Nov 02 21:08:25.100586 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.100553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:25.100586 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.100570    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:25.100788 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.100674    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:25.118607 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.118585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"
Nov 02 21:08:25.155621 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.155594    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 21:08:25.155621 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.155594    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2mhzd\""
Nov 02 21:08:25.155823 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.155718    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnfwz\""
Nov 02 21:08:25.155878 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.155818    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9p4p6\""
Nov 02 21:08:25.155946 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.155929    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 21:08:25.156938 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.156890    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 21:08:25.156938 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.156923    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 21:08:25.279522 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.279479    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n64lc" event={"ID":"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870","Type":"ContainerStarted","Data":"e8f3a26dfe0f75d02835d2804613f920517da798932f531f2a029cdf4a4f4a26"}
Nov 02 21:08:25.292885 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.292843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:25.293063 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.292928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:25.293063 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.292953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:25.293063 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:25.293028    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 21:08:25.293063 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:25.293042    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:25.293312 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:25.293099    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:27.293080031 +0000 UTC m=+35.698105427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : configmap "kube-rbac-proxy" not found
Nov 02 21:08:25.293312 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:25.293130    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:27.293123429 +0000 UTC m=+35.698148822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:25.293312 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:25.293150    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:08:25.293312 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:25.293186    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls podName:178af0c6-6aaf-4a57-9ed4-25988a9bac33 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:27.293171299 +0000 UTC m=+35.698196699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls") pod "dns-default-wf86r" (UID: "178af0c6-6aaf-4a57-9ed4-25988a9bac33") : secret "dns-default-metrics-tls" not found
Nov 02 21:08:25.492596 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.492529    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"]
Nov 02 21:08:25.495746 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:08:25.495717    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7740f0f0_41ba_4325_9a49_1ff07a8c8475.slice/crio-c9b275ec1d0ef79e982bf95191d61477f397411007719391d3bcdffa50eddca6 WatchSource:0}: Error finding container c9b275ec1d0ef79e982bf95191d61477f397411007719391d3bcdffa50eddca6: Status 404 returned error can't find the container with id c9b275ec1d0ef79e982bf95191d61477f397411007719391d3bcdffa50eddca6
Nov 02 21:08:25.897507 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.897471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:25.897806 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:25.897623    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:08:25.897806 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:25.897696    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:57.897677435 +0000 UTC m=+66.302702831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : secret "metrics-daemon-secret" not found
Nov 02 21:08:25.998324 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.998284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:25.998535 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:25.998518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:26.000844 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.000820    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:26.001189 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.001165    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:26.054046 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.054009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:26.067263 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.067237    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:26.283609 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.283566    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f" event={"ID":"7740f0f0-41ba-4325-9a49-1ff07a8c8475","Type":"ContainerStarted","Data":"c9b275ec1d0ef79e982bf95191d61477f397411007719391d3bcdffa50eddca6"}
Nov 02 21:08:26.448804 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.448735    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-plprl"]
Nov 02 21:08:26.489945 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:08:26.489910    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71142023_8714_4106_8db9_e3522b32da9e.slice/crio-54a6b4319de3a8197d4b73038ef6156535ef49a922a1fec4b1adcb723fd8414f WatchSource:0}: Error finding container 54a6b4319de3a8197d4b73038ef6156535ef49a922a1fec4b1adcb723fd8414f: Status 404 returned error can't find the container with id 54a6b4319de3a8197d4b73038ef6156535ef49a922a1fec4b1adcb723fd8414f
Nov 02 21:08:26.495219 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.495083    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zl2hk"]
Nov 02 21:08:26.498837 ip-10-0-133-86 kubenswrapper[2583]: W1102 21:08:26.498797    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9be1c6f_dae7_418e_ae7d_d0cd42578469.slice/crio-e606364ce9f7c8581e43a1e9e3f1c51c6d9b5b2aeba3461cc587494a4a781080 WatchSource:0}: Error finding container e606364ce9f7c8581e43a1e9e3f1c51c6d9b5b2aeba3461cc587494a4a781080: Status 404 returned error can't find the container with id e606364ce9f7c8581e43a1e9e3f1c51c6d9b5b2aeba3461cc587494a4a781080
Nov 02 21:08:26.899291 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.899262    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-748zb"]
Nov 02 21:08:26.926244 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.926219    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:26.956450 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:26.956419    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-748zb"]
Nov 02 21:08:27.003021 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.002441    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ltkgn\""
Nov 02 21:08:27.003021 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.002521    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 21:08:27.003021 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.002554    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 21:08:27.003286 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.003206    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 21:08:27.003286 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.003242    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 21:08:27.108178 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.108143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-key\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.108178 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.108182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-cabundle\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.108425 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.108317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps67r\" (UniqueName: \"kubernetes.io/projected/70276149-d6d6-4728-9ffd-9d9351c23ac6-kube-api-access-ps67r\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.209011 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.208976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-key\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.209011 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.209013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-cabundle\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.209259 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.209057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps67r\" (UniqueName: \"kubernetes.io/projected/70276149-d6d6-4728-9ffd-9d9351c23ac6-kube-api-access-ps67r\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.209791 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.209765    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-cabundle\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.211652 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.211629    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-key\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.264364 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.264330    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps67r\" (UniqueName: \"kubernetes.io/projected/70276149-d6d6-4728-9ffd-9d9351c23ac6-kube-api-access-ps67r\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.287882 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.287850    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zl2hk" event={"ID":"b9be1c6f-dae7-418e-ae7d-d0cd42578469","Type":"ContainerStarted","Data":"e606364ce9f7c8581e43a1e9e3f1c51c6d9b5b2aeba3461cc587494a4a781080"}
Nov 02 21:08:27.289369 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.289330    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-plprl" event={"ID":"71142023-8714-4106-8db9-e3522b32da9e","Type":"ContainerStarted","Data":"54a6b4319de3a8197d4b73038ef6156535ef49a922a1fec4b1adcb723fd8414f"}
Nov 02 21:08:27.309550 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.309525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:27.309645 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.309560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:27.309645 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.309626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:27.309749 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:27.309678    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:27.309749 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:27.309701    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 21:08:27.309749 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:27.309738    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:31.30972079 +0000 UTC m=+39.714746184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:27.309847 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:27.309752    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:31.309745506 +0000 UTC m=+39.714770899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : configmap "kube-rbac-proxy" not found
Nov 02 21:08:27.309847 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:27.309779    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:08:27.309847 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:27.309827    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls podName:178af0c6-6aaf-4a57-9ed4-25988a9bac33 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:31.309814588 +0000 UTC m=+39.714839980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls") pod "dns-default-wf86r" (UID: "178af0c6-6aaf-4a57-9ed4-25988a9bac33") : secret "dns-default-metrics-tls" not found
Nov 02 21:08:27.541927 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.541896    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:27.933617 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:27.933492    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-748zb"]
Nov 02 21:08:28.294848 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:28.294746    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8924h-85j4m" event={"ID":"283a86d8-9e77-4f36-8fa9-0bd380ec043f","Type":"ContainerStarted","Data":"123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9"}
Nov 02 21:08:28.296860 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:28.296792    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f" event={"ID":"7740f0f0-41ba-4325-9a49-1ff07a8c8475","Type":"ContainerStarted","Data":"2dbe310167b804dfa7f1c50fc12069a1052a86266b111459a37f8cce3ae35687"}
Nov 02 21:08:28.298311 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:28.298283    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-748zb" event={"ID":"70276149-d6d6-4728-9ffd-9d9351c23ac6","Type":"ContainerStarted","Data":"9b3e2feab208167edd7fa9189ff7e58f0b6764b5c484d463b4b70f54147a5096"}
Nov 02 21:08:28.324901 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:28.324847    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qzbfv_8444a0ef-d9fe-4605-97e3-6d6263228a11/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:28.406154 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:28.405136    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8924h-85j4m" podStartSLOduration=2.356922815 podStartE2EDuration="5.405096301s" totalImagesPullingTime="3.048173486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:28.405060893 +0000 UTC m=+36.810086308" watchObservedRunningTime="2026-11-02 21:08:28.405096301 +0000 UTC m=+36.810121720"
Nov 02 21:08:28.514910 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:28.514846    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f" podStartSLOduration=2.510536322 podStartE2EDuration="4.514826195s" totalImagesPullingTime="2.004289873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:28.513650202 +0000 UTC m=+36.918675618" watchObservedRunningTime="2026-11-02 21:08:28.514826195 +0000 UTC m=+36.919851607"
Nov 02 21:08:28.609125 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:28.609016    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qzbfv_8444a0ef-d9fe-4605-97e3-6d6263228a11/konnectivity-agent/0.log" limit=1024
Nov 02 21:08:29.305040 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:29.304595    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n64lc" event={"ID":"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870","Type":"ContainerStarted","Data":"351667487cfc82021313ba796fa2b76b05472204a20b37c8b460fac8e4380f1b"}
Nov 02 21:08:29.420144 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:29.419619    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n64lc" podStartSLOduration=2.575660997 podStartE2EDuration="6.41959689s" totalImagesPullingTime="3.843935893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:29.415181751 +0000 UTC m=+37.820207189" watchObservedRunningTime="2026-11-02 21:08:29.41959689 +0000 UTC m=+37.824622305"
Nov 02 21:08:31.311005 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:31.310971    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-plprl" event={"ID":"71142023-8714-4106-8db9-e3522b32da9e","Type":"ContainerStarted","Data":"f3f348a234d1b4970fce173f638a1761a7952f6398bcc5427a08c2051c3f4853"}
Nov 02 21:08:31.345768 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:31.345728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:31.345922 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:31.345777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:31.345922 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:31.345828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:31.345922 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:31.345888    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:08:31.346021 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:31.345962    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls podName:178af0c6-6aaf-4a57-9ed4-25988a9bac33 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:39.345940827 +0000 UTC m=+47.750966240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls") pod "dns-default-wf86r" (UID: "178af0c6-6aaf-4a57-9ed4-25988a9bac33") : secret "dns-default-metrics-tls" not found
Nov 02 21:08:31.346021 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:31.345888    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:31.346021 ip-10-0-133-86 kubenswrapper[2583]: E1102 21:08:31.346010    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:39.345999251 +0000 UTC m=+47.751024680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:31.346457 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:31.346427    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:31.415453 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:31.415425    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zc285_a9941cc1-f486-4fbc-bc93-9b6070db00d3/dns-node-resolver/0.log"
Nov 02 21:08:32.317393 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:32.317353    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-748zb" event={"ID":"70276149-d6d6-4728-9ffd-9d9351c23ac6","Type":"ContainerStarted","Data":"937aabefb68a1c7af917db33fc2825428151951c55c2d172542fd34e720bdc37"}
Nov 02 21:08:32.415663 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:32.415634    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sllxw_d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f/node-ca/0.log"
Nov 02 21:08:32.426231 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:32.426182    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-748zb" podStartSLOduration=3.152446675 podStartE2EDuration="6.426153251s" totalImagesPullingTime="3.273706576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:32.425227724 +0000 UTC m=+40.830253155" watchObservedRunningTime="2026-11-02 21:08:32.426153251 +0000 UTC m=+40.831178667"
Nov 02 21:08:32.535754 ip-10-0-133-86 kubenswrapper[2583]: I1102 21:08:32.535704    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-plprl" podStartSLOduration=35.827646313 podStartE2EDuration="40.535690679s" totalImagesPullingTime="4.708044366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:32.535571221 +0000 UTC m=+40.940596635" watchObservedRunningTime="2026-11-02 21:08:32.535690679 +0000 UTC m=+40.940716093"
Nov 02 21:08:32.677197 ip-10-0-133-86 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 21:08:32.692488 ip-10-0-133-86 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 21:08:32.692722 ip-10-0-133-86 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 21:08:32.692789 ip-10-0-133-86 systemd[1]: kubelet.service: Consumed 2.485s CPU time.
Nov 02 21:08:32.700791 ip-10-0-133-86 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 21:08:32.805500 ip-10-0-133-86 kubenswrapper[4528]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:08:32.805500 ip-10-0-133-86 kubenswrapper[4528]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 21:08:32.805500 ip-10-0-133-86 kubenswrapper[4528]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:08:32.805500 ip-10-0-133-86 kubenswrapper[4528]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 21:08:32.807825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807732    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:08:32.807825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807819    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:08:32.807825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807824    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:08:32.807825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807827    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:08:32.807825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807830    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:08:32.807825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807832    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:08:32.807825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807835    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807839    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807841    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807846    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807850    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807853    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807856    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807858    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807861    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807864    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807867    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807870    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807872    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807875    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807878    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807882    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807884    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807887    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807889    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:08:32.808087 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807892    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807894    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807897    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807900    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807902    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807904    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807908    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807910    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807913    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807916    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807918    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807920    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807923    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807926    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807928    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807931    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807933    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807936    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807938    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:08:32.808560 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807941    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807943    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807946    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807949    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807951    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807955    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807960    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807963    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807966    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807969    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807972    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807974    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807977    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807979    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807982    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807984    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807987    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807989    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807992    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:08:32.809081 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807995    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.807998    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808001    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808003    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808006    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808008    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808011    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808013    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808017    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808019    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808021    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808024    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808026    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808029    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808031    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808034    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808037    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808039    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808042    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808044    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:08:32.809581 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808047    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808049    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808051    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808054    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808057    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808059    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808062    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808065    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808067    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808070    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808073    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808075    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808079    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808081    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808084    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808087    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808089    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808092    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808095    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:08:32.810088 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808098    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808101    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808103    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808123    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808880    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808888    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808891    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808894    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808897    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808899    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808902    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808905    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808926    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808929    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808932    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808934    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808937    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808940    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808943    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808946    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:08:32.810559 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808949    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808952    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808954    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808957    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808962    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808968    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808972    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808974    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808978    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808980    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808983    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808986    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808989    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808991    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808994    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.808997    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809000    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809003    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809005    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:08:32.811216 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809008    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809010    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809013    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809016    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809019    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809021    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809024    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809028    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809030    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809033    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809036    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809039    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809042    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809045    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809047    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809050    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809053    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809055    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809058    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:08:32.811867 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809061    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809063    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809066    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809069    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809071    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809074    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809077    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809079    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809082    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809085    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809088    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809091    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809094    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809098    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809101    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809104    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809120    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809123    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809126    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809129    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:08:32.812337 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809132    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809135    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809138    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809140    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809143    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809146    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809149    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809152    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809154    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809157    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809160    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809163    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809167    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809171    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809174    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809177    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809180    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809183    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809186    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809189    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:08:32.812831 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809191    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809194    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809197    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809201    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809205    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809208    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809210    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809213    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809215    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809218    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809220    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809223    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809706    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809710    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809713    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809716    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809719    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809724    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809728    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:08:32.813368 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809731    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809735    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809738    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809741    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809743    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809746    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809748    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809754    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809757    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809760    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809763    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809766    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809769    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809772    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809775    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809777    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809781    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809783    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809786    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809789    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:08:32.813865 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809791    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809794    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809797    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809799    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809802    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809804    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809807    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809809    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809812    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809815    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809817    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809820    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809823    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809825    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809828    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809831    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809833    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809836    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809840    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:08:32.814411 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809844    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809847    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809850    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809853    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809855    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809858    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809860    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809863    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809866    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809868    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809871    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809873    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809876    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809880    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809883    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809885    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809888    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809891    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809894    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:08:32.814950 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809896    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809898    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809901    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809904    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809906    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809909    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809911    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809915    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809917    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809920    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809922    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809925    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809928    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809932    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809935    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809937    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809941    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809944    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809947    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:08:32.815422 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809950    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809953    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809956    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809959    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809962    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809964    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809967    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809970    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809972    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809975    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809978    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809981    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809983    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809986    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809988    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809991    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.809998    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.810001    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.810004    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.810006    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:08:32.815888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.810009    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.810012    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810148    4528 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810159    4528 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810166    4528 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810171    4528 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810177    4528 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810181    4528 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810185    4528 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810189    4528 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810193    4528 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810196    4528 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810199    4528 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810203    4528 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810206    4528 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810209    4528 flags.go:64] FLAG: --cgroup-root=""
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810212    4528 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810215    4528 flags.go:64] FLAG: --client-ca-file=""
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810218    4528 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810221    4528 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810224    4528 flags.go:64] FLAG: --cluster-domain=""
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810227    4528 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810230    4528 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810233    4528 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 21:08:32.816393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810236    4528 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810240    4528 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810244    4528 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810247    4528 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810250    4528 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810253    4528 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810256    4528 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810259    4528 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810262    4528 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810265    4528 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810270    4528 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810273    4528 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810276    4528 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810279    4528 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810284    4528 flags.go:64] FLAG: --enable-server="true"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810287    4528 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810291    4528 flags.go:64] FLAG: --event-burst="100"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810294    4528 flags.go:64] FLAG: --event-qps="50"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810297    4528 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810300    4528 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810302    4528 flags.go:64] FLAG: --eviction-hard=""
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810307    4528 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810309    4528 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810313    4528 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810316    4528 flags.go:64] FLAG: --eviction-soft=""
Nov 02 21:08:32.817101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810319    4528 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810322    4528 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810324    4528 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810328    4528 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810331    4528 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810333    4528 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810336    4528 flags.go:64] FLAG: --feature-gates=""
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810340    4528 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810343    4528 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810347    4528 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810351    4528 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810354    4528 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810357    4528 flags.go:64] FLAG: --help="false"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810360    4528 flags.go:64] FLAG: --hostname-override="ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810364    4528 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810366    4528 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810369    4528 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810373    4528 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810377    4528 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810380    4528 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810383    4528 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810387    4528 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810390    4528 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 21:08:32.817758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810393    4528 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810396    4528 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810399    4528 flags.go:64] FLAG: --kube-reserved=""
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810402    4528 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810405    4528 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810408    4528 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810411    4528 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810414    4528 flags.go:64] FLAG: --lock-file=""
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810417    4528 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810420    4528 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810423    4528 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810428    4528 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810431    4528 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810434    4528 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810437    4528 flags.go:64] FLAG: --logging-format="text"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810440    4528 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810444    4528 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810447    4528 flags.go:64] FLAG: --manifest-url=""
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810452    4528 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810456    4528 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810459    4528 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810463    4528 flags.go:64] FLAG: --max-pods="110"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810466    4528 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810469    4528 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810473    4528 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 21:08:32.818337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810476    4528 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810479    4528 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810482    4528 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810486    4528 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810493    4528 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810497    4528 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810501    4528 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810504    4528 flags.go:64] FLAG: --pod-cidr=""
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810507    4528 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810510    4528 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810513    4528 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810515    4528 flags.go:64] FLAG: --port="10250"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810519    4528 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810522    4528 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0004c62d18ace5d95"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810525    4528 flags.go:64] FLAG: --qos-reserved=""
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810528    4528 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810531    4528 flags.go:64] FLAG: --register-node="true"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810533    4528 flags.go:64] FLAG: --register-with-taints=""
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810537    4528 flags.go:64] FLAG: --registry-burst="10"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810540    4528 flags.go:64] FLAG: --registry-qps="5"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810543    4528 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810546    4528 flags.go:64] FLAG: --reserved-memory=""
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810549    4528 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810553    4528 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810556    4528 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810560    4528 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 21:08:32.818993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810564    4528 flags.go:64] FLAG: --runonce="false"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810567    4528 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810570    4528 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810573    4528 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810576    4528 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810579    4528 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810582    4528 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810585    4528 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810589    4528 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810591    4528 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810595    4528 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810599    4528 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810601    4528 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810605    4528 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810607    4528 flags.go:64] FLAG: --system-cgroups=""
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810610    4528 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810615    4528 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810618    4528 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810621    4528 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810624    4528 flags.go:64] FLAG: --tls-min-version=""
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810627    4528 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810630    4528 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810632    4528 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810636    4528 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810639    4528 flags.go:64] FLAG: --v="2"
Nov 02 21:08:32.819701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810643    4528 flags.go:64] FLAG: --version="false"
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810647    4528 flags.go:64] FLAG: --vmodule=""
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810651    4528 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.810654    4528 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811284    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811299    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811306    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811312    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811317    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811321    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811326    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811330    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811335    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811344    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811348    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811353    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811357    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811361    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811367    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811371    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:08:32.820335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811375    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811379    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811383    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811388    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811392    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811396    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811405    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811409    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811413    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811417    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811421    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811425    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811429    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811433    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811437    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811442    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811446    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811450    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811457    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:08:32.820980 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811465    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811470    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811474    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811478    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811482    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811486    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811491    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811495    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811499    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811504    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811508    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811512    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811522    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811527    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811531    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811535    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811540    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811544    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811548    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:08:32.821602 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811552    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811556    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811560    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811564    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811568    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811572    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811580    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811584    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811590    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811596    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811601    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811606    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811612    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811616    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811620    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811625    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811635    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811640    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811645    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:08:32.822334 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811649    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811652    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811655    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811658    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811661    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811666    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811669    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811672    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811675    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811678    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811685    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811689    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811693    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811726    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811866    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811873    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811876    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811880    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811883    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:08:32.822827 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811886    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811889    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811892    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811897    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811900    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811903    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811906    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811909    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811912    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811915    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811918    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811921    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811923    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.811926    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.811968    4528 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812013    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812018    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:08:32.823393 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812022    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812025    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812028    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812032    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812035    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812038    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812041    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812044    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812047    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812050    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812053    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812055    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812058    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812071    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812075    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812077    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812080    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812083    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812085    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:08:32.823798 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812089    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812092    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812095    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812097    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812100    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812102    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812120    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812125    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812129    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812134    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812138    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812143    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812147    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812151    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812155    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812159    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812163    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812167    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812175    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812179    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:08:32.824370 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812183    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812187    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812192    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812195    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812197    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812200    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812204    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812209    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812212    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812215    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812218    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812220    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812223    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812226    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812229    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812232    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812235    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812237    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812240    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:08:32.825073 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812243    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812246    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812249    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812251    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812254    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812257    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812259    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812262    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812267    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812272    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812277    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812281    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812284    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812287    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812290    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812293    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812295    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812298    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812300    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:08:32.825612 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812303    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812306    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812308    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812311    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812313    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812316    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812319    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812322    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812325    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812327    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812330    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812333    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812336    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812338    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812341    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812343    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812346    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812349    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812352    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812355    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:08:32.826092 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812357    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812360    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812363    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812367    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812372    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812374    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.812377    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.816481    4528 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.816497    4528 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.816515    4528 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.816519    4528 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816573    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816579    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816583    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816587    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816590    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816594    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816596    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816599    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:08:32.826634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816602    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816604    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816607    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816609    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816612    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816615    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816617    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816620    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816623    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816625    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816628    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816631    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816634    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816636    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816639    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816641    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816644    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816647    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816650    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816652    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:08:32.827342 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816655    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816658    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816661    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816664    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816666    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816669    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816672    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816674    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816677    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816679    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816682    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816684    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816687    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816690    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816692    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816696    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816698    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816701    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816703    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:08:32.828199 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816706    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816709    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816711    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816714    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816716    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816719    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816721    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816724    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816727    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816730    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816732    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816735    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816738    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816740    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816744    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816748    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816752    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816755    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816758    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:08:32.828825 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816760    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816763    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816765    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816768    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816770    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816773    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816776    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816779    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816782    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816785    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816787    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816790    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816793    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816795    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816798    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816801    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816803    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816807    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816809    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816812    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:08:32.829503 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816815    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816818    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816820    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816823    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816826    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816828    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816831    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816834    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816837    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816840    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816843    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816846    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816849    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816851    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816854    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816856    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816859    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816863    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816867    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:08:32.830142 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.816878    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.816924    4528 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817070    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817075    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817078    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817081    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817084    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817087    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817090    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817093    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817096    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817099    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817101    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817104    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817128    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817132    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:08:32.830736 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817135    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817137    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817140    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817144    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817148    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817151    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817154    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817156    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817160    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817163    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817165    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817168    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817171    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817173    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817178    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817181    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817184    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817187    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817190    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:08:32.831251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817192    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817195    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817198    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817201    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817203    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817206    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817209    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817211    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817214    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817217    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817219    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817222    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817225    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817227    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817230    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817233    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817235    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817238    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817240    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:08:32.831714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817243    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817246    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817249    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817253    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817255    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817258    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817261    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817263    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817266    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817268    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817271    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817273    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817277    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817280    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817282    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817285    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817288    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817290    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817293    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:08:32.832286 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817296    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817298    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817301    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817304    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817306    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817309    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817312    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817315    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817318    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817320    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817323    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817329    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817332    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817334    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817337    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817340    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817343    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817346    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817348    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817351    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:08:32.832842 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817354    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817357    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817359    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817362    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817365    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817367    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817370    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817373    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817375    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817395    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817400    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817403    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817406    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817409    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817412    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.817436    4528 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 21:08:32.833743 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817467    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817471    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817474    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817477    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817479    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817482    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817484    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817489    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817492    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817495    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817497    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817500    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817503    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817505    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817509    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817511    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817514    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817517    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817520    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817522    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 21:08:32.834281 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817525    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817528    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817530    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817533    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817536    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817539    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817541    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817544    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817546    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817549    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817551    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817554    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817557    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817559    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817562    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817565    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817567    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817570    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817572    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817576    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 21:08:32.834829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817579    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817582    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817584    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817587    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817589    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817592    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817595    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817598    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817601    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817604    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817606    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817609    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817611    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817615    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817618    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817621    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817624    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817627    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817629    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 21:08:32.835346 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817632    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817635    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817637    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817640    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817643    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817647    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817651    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817654    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817657    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817659    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817662    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817664    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817669    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817671    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817674    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817677    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817679    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817682    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817684    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 21:08:32.835828 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817687    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817690    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817696    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817699    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817702    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817705    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817707    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817710    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817712    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817715    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817718    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817721    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817723    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817726    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817729    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817731    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817734    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817736    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817739    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817742    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 21:08:32.836304 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817744    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817747    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817749    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817752    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817754    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817758    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817762    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:32.817764    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.817817    4528 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.818828    4528 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.818880    4528 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.819072    4528 server.go:1014] "Starting client certificate rotation"
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.819211    4528 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.819267    4528 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:02:52 +0000 UTC" deadline="2028-07-21 04:39:13.798143344 +0000 UTC"
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.819301    4528 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15031h30m40.978843928s"
Nov 02 21:08:32.836815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.819572    4528 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 21:08:32.837241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.820799    4528 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 21:08:32.837241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.822399    4528 log.go:25] "Validated CRI v1 runtime API"
Nov 02 21:08:32.837241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.823989    4528 log.go:25] "Validated CRI v1 image API"
Nov 02 21:08:32.837241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.824762    4528 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 21:08:32.837241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.826562    4528 fs.go:135] Filesystem UUIDs: map[3f364eca-ea43-4346-b859-733d9a639e1e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eeb97604-fe3d-4c55-b35f-937ee5c23d19:/dev/nvme0n1p3]
Nov 02 21:08:32.837440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.826584    4528 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cf460d885ddd8bff4394a61fbee03ac6b7d9f3f3fa6dd8ceaeaeaaa254b3df3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cf460d885ddd8bff4394a61fbee03ac6b7d9f3f3fa6dd8ceaeaeaaa254b3df3/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36a600f3c16e2abc954257579b18aceec4eda40e85b5e8fcb038c0ed9f49c0f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36a600f3c16e2abc954257579b18aceec4eda40e85b5e8fcb038c0ed9f49c0f6/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e2cd58226b177797b2793a400a896aa649159db0db4f54656bb6475f31e6911/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e2cd58226b177797b2793a400a896aa649159db0db4f54656bb6475f31e6911/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54a6b4319de3a8197d4b73038ef6156535ef49a922a1fec4b1adcb723fd8414f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54a6b4319de3a8197d4b73038ef6156535ef49a922a1fec4b1adcb723fd8414f/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5890923407df8992f3d0e8dacf7800593f826b4adb22ff1df4933a7c75582188/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5890923407df8992f3d0e8dacf7800593f826b4adb22ff1df4933a7c75582188/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6159573cd2493cc31b2978e50b973dd37d4a41c47a2699229ee62da99618d6e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6159573cd2493cc31b2978e50b973dd37d4a41c47a2699229ee62da99618d6e9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ec3711a66e817b4bce816161483d39b166c56a5145291e10214e65bfb56f885/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ec3711a66e817b4bce816161483d39b166c56a5145291e10214e65bfb56f885/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83cddc74cea102b6aa974c9c5c094dee54bc7b01e929cdfaf219c2cf600343dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83cddc74cea102b6aa974c9c5c094dee54bc7b01e929cdfaf219c2cf600343dc/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b3e2feab208167edd7fa9189ff7e58f0b6764b5c484d463b4b70f54147a5096/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b3e2feab208167edd7fa9189ff7e58f0b6764b5c484d463b4b70f54147a5096/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6002d2a8301d23f4894c58372942988e309f1ca0484e056bc56b68311903650/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6002d2a8301d23f4894c58372942988e309f1ca0484e056bc56b68311903650/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9b275ec1d0ef79e982bf95191d61477f397411007719391d3bcdffa50eddca6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9b275ec1d0ef79e982bf95191d61477f397411007719391d3bcdffa50eddca6/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3e9926d4be3dcd486d78f7b6a6d146e486b71c4bf81a33f1349044e895298f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3e9926d4be3dcd486d78f7b6a6d146e486b71c4bf81a33f1349044e895298f4/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e606364ce9f7c8581e43a1e9e3f1c51c6d9b5b2aeba3461cc587494a4a781080/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e606364ce9f7c8581e43a1e9e3f1c51c6d9b5b2aeba3461cc587494a4a781080/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8f3a26dfe0f75d02835d2804613f920517da798932f531f2a029cdf4a4f4a26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8f3a26dfe0f75d02835d2804613f920517da798932f531f2a029cdf4a4f4a26/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f011932736ec69596fe0ab65960926916017f33d73a9613ba88fae4b53e21691/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f011932736ec69596fe0ab65960926916017f33d73a9613ba88fae4b53e21691/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5269b8e1bb3da3ee4d156f7758aa21e85a02dd3940b057c9b99aa59b14c48c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5269b8e1bb3da3ee4d156f7758aa21e85a02dd3940b057c9b99aa59b14c48c0/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870/volumes/kubernetes.io~projected/kube-api-access-9tgq4:{mountpoint:/var/lib/kubelet/pods/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870/volumes/kubernetes.io~projected/kube-api-access-9tgq4 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12e057a9-159b-465f-9864-3d4521016ac6/volumes/kubernetes.io~projected/kube-api-access-r9hr2:{mountpoint:/var/lib/kubelet/pods/12e057a9-159b-465f-9864-3d4521016ac6/volumes/kubernetes.io~projected/kube-api-access-r9hr2 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/178af0c6-6aaf-4a57-9ed4-25988a9bac33/volumes/kubernetes.io~projected/kube-api-access-jn6h5:{mountpoint:/var/lib/kubelet/pods/178af0c6-6aaf-4a57-9ed4-25988a9bac33/volumes/kubernetes.io~projected/kube-api-access-jn6h5 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b38e492-9c4b-4190-8456-8a2b965ad3e7/volumes/kubernetes.io~projected/kube-api-access-9vgmh:{mountpoint:/var/lib/kubelet/pods/1b38e492-9c4b-4190-8456-8a2b965ad3e7/volumes/kubernetes.io~projected/kube-api-access-9vgmh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/211397c2-9e2b-48fe-9fb4-050a6cba9a5c/volumes/kubernetes.io~projected/kube-api-access-89fr2:{mountpoint:/var/lib/kubelet/pods/211397c2-9e2b-48fe-9fb4-050a6cba9a5c/volumes/kubernetes.io~projected/kube-api-access-89fr2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/283a86d8-9e77-4f36-8fa9-0bd380ec043f/volumes/kubernetes.io~projected/kube-api-access-njx2v:{mountpoint:/var/lib/kubelet/pods/283a86d8-9e77-4f36-8fa9-0bd380ec043f/volumes/kubernetes.io~projected/kube-api-access-njx2v major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60d81696-4204-4290-81f7-8cfe42e3bed0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/60d81696-4204-4290-81f7-8cfe42e3bed0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60d81696-4204-4290-81f7-8cfe42e3bed0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/60d81696-4204-4290-81f7-8cfe42e3bed0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60d81696-4204-4290-81f7-8cfe42e3bed0/volumes/kubernetes.io~projected/kube-api-access-ddwwx:{mountpoint:/var/lib/kubelet/pods/60d81696-4204-4290-81f7-8cfe42e3bed0/volumes/kubernetes.io~projected/kube-api-access-ddwwx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70276149-d6d6-4728-9ffd-9d9351c23ac6/volumes/kubernetes.io~projected/kube-api-access-ps67r:{mountpoint:/var/lib/kubelet/pods/70276149-d6d6-4728-9ffd-9d9351c23ac6/volumes/kubernetes.io~projected/kube-api-access-ps67r major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70276149-d6d6-4728-9ffd-9d9351c23ac6/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/70276149-d6d6-4728-9ffd-9d9351c23ac6/volumes/kubernetes.io~secret/signing-key major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71142023-8714-4106-8db9-e3522b32da9e/volumes/kubernetes.io~projected/kube-api-access-t8lmb:{mountpoint:/var/lib/kubelet/pods/71142023-8714-4106-8db9-e3522b32da9e/volumes/kubernetes.io~projected/kube-api-access-t8lmb major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7740f0f0-41ba-4325-9a49-1ff07a8c8475/volumes/kubernetes.io~projected/kube-api-access-m6qmr:{mountpoint:/var/lib/kubelet/pods/7740f0f0-41ba-4325-9a49-1ff07a8c8475/volumes/kubernetes.io~projected/kube-api-access-m6qmr major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8444a0ef-d9fe-4605-97e3-6d6263228a11/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8444a0ef-d9fe-4605-97e3-6d6263228a11/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8abc6131-3792-4669-bfe7-a1c88ee317a3/volumes/kubernetes.io~projected/kube-api-access-stph2:{mountpoint:/var/lib/kubelet/pods/8abc6131-3792-4669-bfe7-a1c88ee317a3/volumes/kubernetes.io~projected/kube-api-access-stph2 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9118c60c-2783-4f6e-8f31-2f5bd6f8d684/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9118c60c-2783-4f6e-8f31-2f5bd6f8d684/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9118c60c-2783-4f6e-8f31-2f5bd6f8d684/volumes/kubernetes.io~projected/kube-api-access-2scqv:{mountpoint:/var/lib/kubelet/pods/9118c60c-2783-4f6e-8f31-2f5bd6f8d684/volumes/kubernetes.io~projected/kube-api-access-2scqv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9118c60c-2783-4f6e-8f31-2f5bd6f8d684/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9118c60c-2783-4f6e-8f31-2f5bd6f8d684/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9941cc1-f486-4fbc-bc93-9b6070db00d3/volumes/kubernetes.io~projected/kube-api-access-f2vm8:{mountpoint:/var/lib/kubelet/pods/a9941cc1-f486-4fbc-bc93-9b6070db00d3/volumes/kubernetes.io~projected/kube-api-access-f2vm8 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a/volumes/kubernetes.io~projected/kube-api-access-xwzls:{mountpoint:/var/lib/kubelet/pods/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a/volumes/kubernetes.io~projected/kube-api-access-xwzls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9be1c6f-dae7-418e-ae7d-d0cd42578469/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b9be1c6f-dae7-418e-ae7d-d0cd42578469/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c/volumes/kubernetes.io~projected/kube-api-access-2pjs7:{mountpoint:/var/lib/kubelet/pods/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c/volumes/kubernetes.io~projected/kube-api-access-2pjs7 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f/volumes/kubernetes.io~projected/kube-api-access-6zs6t:{mountpoint:/var/lib/kubelet/pods/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f/volumes/kubernetes.io~projected/kube-api-access-6zs6t major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/536c385d775b716d38ce3595e1bb0cde4f6bbc75eb93c0b6509cc17729daff8a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/054a7b989de00bbd5ed78d07052396d2734c4d78105a7669e14d5c394396ea24/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5d925e6561659b48cba5fa0cfffa822d6f691362791e38c13c26e6af54dafea7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/da24db675089471894597f88a407c7661f57ed55722c872852e643684449695e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b13d6c07684dacccd4ad630c988c02fd5aa125e80606fee1cb42ce2fcaffb1f4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f6975bb0e1d187ce0f0ec2fae3a4e44bb1b7e4a8ba259c470e6416da1ff9629b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/91902e7a0a5d527e7dded13b19d9db4032cd7566e52451cab34d785003cfe487/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/93953c02920cf0f3a7e1068a7f21cdb290b31023bdd8af60243e42930007ad65/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/4a095154ac8f8be18a4d0a4083dea566094f0146a3f4ae6025b4a5b60a626de7/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9b22b1089dacfb7e817c564d427c38ea2f2404989337676be88d949453d0c027/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/1c5434a5a844d1fb73722cd703adec0e62b7dc8253c5ebdfd26e2e031ceb30d1/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/bf474857e6305adc7ec1349c0f92877b16532dd4511506740b485fac03cfac49/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6ecef01805d23daaaa65c7e347c1e73d544006254ed749bbc25930b35e12c5c0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/fca207596f29c7f788e71643904056dbb88af63748bee7e116d6e6bea0cb445b/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/de045003efaa494dda84dcc3a11610e866ed17cafae16eb814fabd67c1b935fd/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/cf829e3c03258451b223b76ffe568da55cf546102b8e10786cd832a85f6950cd/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/0308f21edf9a7815aa370c9c1d7b45e9f0ec29dadca6f2323aa134bc011bdbfc/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/eed86da90119db0e92d8b96a691fc53a7229e3e27da9de1f2598bd84533669e0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dfeb940e9e293a1fb55b4309c934580addbe1ccc92352ff7ecf4eff9b2c1086f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9d4e15e043ca4157c560d9ea1770cd5f9b15713b8fb0ea07742649c0455367dd/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/19915b5a7ffef2dace8bb5c3f732e2614cada663ff242d03d63718febdec5367/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/27615e98da45a0c2f55699aed864cb1acb857cc0667c73cc0422501aad115027/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/088e6ec0b0331dac72b8898760001cf37b00b57ff5dc8044e9dee98d252b17cd/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/6e79f53329aca5c60e5b149965d870415bc0b669dabfae6251c51950f76c0b60/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/bd348e9d18e85c5353f5524e14606936a5db726f8d9a1a2be8360a2e0c21b38d/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/25f20b857a18535286edc1e517f339d6b7e8c1e89dc510bdeb468fc786cf3673/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/1a918a5e5e29cc5384dd37d5cd03d13b2efedefa14fec885f6bdc6acf51e0c1d/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/a9233f5e45ea24520cb2ad4653deec50a971c871440f376904ef43b15bf387dc/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/2f392311828058ce9bbea0ebbd0bc4df15dd177bf4fcface629f5fd93d6126e0/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/c88327645d1db67e4569e5f18b4f63839516737a1e802fb2a8177cc0b41e3b7e/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/e2ea32ceb0a04ce4d7e6c7cddb0fbc561fbdb668b69ae383ed78e5b53ad01bd4/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/d5137d835c3b984bed7837344eeba87eb782d0280188cdf157547dac2f8b6d63/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/9aa57d57a0b64e92442d9e7c85555d2e698cc8fa6aeeff9f91812474f37e4e97/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5ad8e82b12dc520b0f7833baacc3a3df87fe2b19b56f035f8ec491d955f83130/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6cc1df13a1971a9708a656d588fed57d5296fd04c6086d4816b254a46bada1ea/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/129d69e7175e2dd1899eb456268b795fdffb4b964d3d65250b09e4076dcb0cc7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3060d9cf7318633b0b66eaef2de62f385e4be6661425b76d6fb2ac4cf9bbc26c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2df35a2f6334b1a96c7780ec48baf14238055e57bb4bc758614cea3539b0fe1f/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3457fc8358c3b1a1d5a319762e65318ab74bd536e606e566890ef603ce4d62e3/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3f2f65ce96189e58a4b2792f57bd50b75bd67252c7f76c265f76a7bb955c5435/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/592ec6f3ec8d63335361d7a68f1e37dd7aacead233e5139237df4cc8aab9d06f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ebc3763d47305b63e80e8729de29ba286071746ce66f1a1e734cec307b29cf93/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e8652e8c2d1fef3a0d187541cb50b1b35a1e5452e60a4da319d5c8282e292cc2/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.835119    4528 manager.go:217] Machine: {Timestamp:2026-11-02 21:08:32.834629161 +0000 UTC m=+0.098165469 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199402 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af6ffeba7bd69ee73c460b87dc911 SystemUUID:ec2af6ff-eba7-bd69-ee73-c460b87dc911 BootID:c8b54ec4-217c-4fc9-a6e4-8c8f904e4526 Filesystems:[{Device:/run/containers/storage/overlay-containers/83cddc74cea102b6aa974c9c5c094dee54bc7b01e929cdfaf219c2cf600343dc/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60d81696-4204-4290-81f7-8cfe42e3bed0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5890923407df8992f3d0e8dacf7800593f826b4adb22ff1df4933a7c75582188/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9b275ec1d0ef79e982bf95191d61477f397411007719391d3bcdffa50eddca6/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54a6b4319de3a8197d4b73038ef6156535ef49a922a1fec4b1adcb723fd8414f/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/70276149-d6d6-4728-9ffd-9d9351c23ac6/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f/volumes/kubernetes.io~projected/kube-api-access-6zs6t DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3e9926d4be3dcd486d78f7b6a6d146e486b71c4bf81a33f1349044e895298f4/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/12e057a9-159b-465f-9864-3d4521016ac6/volumes/kubernetes.io~projected/kube-api-access-r9hr2 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b9be1c6f-dae7-418e-ae7d-d0cd42578469/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/71142023-8714-4106-8db9-e3522b32da9e/volumes/kubernetes.io~projected/kube-api-access-t8lmb DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60d81696-4204-4290-81f7-8cfe42e3bed0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a9941cc1-f486-4fbc-bc93-9b6070db00d3/volumes/kubernetes.io~projected/kube-api-access-f2vm8 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f011932736ec69596fe0ab65960926916017f33d73a9613ba88fae4b53e21691/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/60d81696-4204-4290-81f7-8cfe42e3bed0/volumes/kubernetes.io~projected/kube-api-access-ddwwx DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8abc6131-3792-4669-bfe7-a1c88ee317a3/volumes/kubernetes.io~projected/kube-api-access-stph2 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870/volumes/kubernetes.io~projected/kube-api-access-9tgq4 DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e606364ce9f7c8581e43a1e9e3f1c51c6d9b5b2aeba3461cc587494a4a781080/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b38e492-9c4b-4190-8456-8a2b965ad3e7/volumes/kubernetes.io~projected/kube-api-access-9vgmh DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9118c60c-2783-4f6e-8f31-2f5bd6f8d684/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/283a86d8-9e77-4f36-8fa9-0bd380ec043f/volumes/kubernetes.io~projected/kube-api-access-njx2v DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8f3a26dfe0f75d02835d2804613f920517da798932f531f2a029cdf4a4f4a26/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9118c60c-2783-4f6e-8f31-2f5bd6f8d684/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9118c60c-2783-4f6e-8f31-2f5bd6f8d684/volumes/kubernetes.io~projected/kube-api-access-2scqv DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6159573cd2493cc31b2978e50b973dd37d4a41c47a2699229ee62da99618d6e9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/178af0c6-6aaf-4a57-9ed4-25988a9bac33/volumes/kubernetes.io~projected/kube-api-access-jn6h5 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7740f0f0-41ba-4325-9a49-1ff07a8c8475/volumes/kubernetes.io~projected/kube-api-access-m6qmr DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36a600f3c16e2abc954257579b18aceec4eda40e85b5e8fcb038c0ed9f49c0f6/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5269b8e1bb3da3ee4d156f7758aa21e85a02dd3940b057c9b99aa59b14c48c0/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cf460d885ddd8bff4394a61fbee03ac6b7d9f3f3fa6dd8ceaeaeaaa254b3df3/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e2cd58226b177797b2793a400a896aa649159db0db4f54656bb6475f31e6911/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b3e2feab208167edd7fa9189ff7e58f0b6764b5c484d463b4b70f54147a5096/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8444a0ef-d9fe-4605-97e3-6d6263228a11/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/211397c2-9e2b-48fe-9fb4-050a6cba9a5c/volumes/kubernetes.io~projected/kube-api-access-89fr2 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6002d2a8301d23f4894c58372942988e309f1ca0484e056bc56b68311903650/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c/volumes/kubernetes.io~projected/kube-api-access-2pjs7 DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ec3711a66e817b4bce816161483d39b166c56a5145291e10214e65bfb56f885/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70276149-d6d6-4728-9ffd-9d9351c23ac6/volumes/kubernetes.io~projected/kube-api-access-ps67r DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a/volumes/kubernetes.io~projected/kube-api-access-xwzls DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:54a6b4319de3a81 MacAddress:3a:b6:4c:3e:cc:59 Speed:10000 Mtu:8901} {Name:9b3e2feab208167 MacAddress:26:7b:41:9b:3b:06 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:dd:75:ff:7f Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:e0:ac:f6:76:a3 Speed:0 Mtu:8901} {Name:c9b275ec1d0ef79 MacAddress:52:67:91:66:2e:d5 Speed:10000 Mtu:8901} {Name:e3e9926d4be3dcd MacAddress:82:2f:d5:47:02:00 Speed:10000 Mtu:8901} {Name:e606364ce9f7c85 MacAddress:ce:11:78:9b:a5:25 Speed:10000 Mtu:8901} {Name:e8f3a26dfe0f75d MacAddress:32:77:3d:4c:27:03 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:dd:75:ff:7f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:12:f0:d4:a6:69:68 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:95:73:34:3d:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.835397    4528 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.835444    4528 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.835720    4528 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.835748    4528 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-86.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.835873    4528 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.835881    4528 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.835894    4528 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.835914    4528 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.836002    4528 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.836179    4528 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.836292    4528 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.836326    4528 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.836339    4528 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.836349    4528 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.836358    4528 kubelet.go:405] "Adding apiserver pod source"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.836368    4528 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.837030    4528 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 21:08:32.837657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.837043    4528 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 21:08:32.838382 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.837669    4528 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 21:08:32.840315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840302    4528 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 21:08:32.840315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840315    4528 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 21:08:32.840422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840413    4528 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 21:08:32.840634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840623    4528 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 21:08:32.840663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840651    4528 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 21:08:32.840748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840740    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 21:08:32.840776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840754    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 21:08:32.840776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840761    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 21:08:32.840776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840766    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 21:08:32.840776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840772    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 21:08:32.840878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840778    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 21:08:32.840878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840810    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 21:08:32.840878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840816    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 21:08:32.840878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840823    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 21:08:32.840878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840829    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 21:08:32.840878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840843    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 21:08:32.840878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840853    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 21:08:32.841184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840927    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 21:08:32.841184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.840934    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 21:08:32.844406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.844395    4528 server.go:1263] "Started kubelet"
Nov 02 21:08:32.844561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.844463    4528 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 21:08:32.844631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.844555    4528 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 21:08:32.844631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.844627    4528 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 21:08:32.845012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.844991    4528 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 21:08:32.845055 ip-10-0-133-86 systemd[1]: Started Kubernetes Kubelet.
Nov 02 21:08:32.847335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.847298    4528 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 21:08:32.849274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.849254    4528 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 21:08:32.850006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.849907    4528 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 21:08:32.850006 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:32.849953    4528 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:08:32.850006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.849969    4528 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 21:08:32.851652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.851630    4528 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 21:08:32.851744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.849956    4528 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:02:54 +0000 UTC" deadline="2028-06-26 05:18:00.150359127 +0000 UTC"
Nov 02 21:08:32.851744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.851710    4528 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14432h9m27.298652566s"
Nov 02 21:08:32.851744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.851190    4528 factory.go:55] Registering systemd factory
Nov 02 21:08:32.851744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.851733    4528 factory.go:223] Registration of the systemd container factory successfully
Nov 02 21:08:32.851923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.850501    4528 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 21:08:32.853184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853144    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c11c4c6b-5bf0-4203-b5c6-2287c0ff122c" volumeName="kubernetes.io/projected/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kube-api-access-2pjs7" seLinuxMountContext=""
Nov 02 21:08:32.853184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853182    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b8cdafe-c2f7-433c-aac4-f4c8c6adf870" volumeName="kubernetes.io/secret/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-cert" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853192    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b38e492-9c4b-4190-8456-8a2b965ad3e7" volumeName="kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-daemon-config" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853200    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b38e492-9c4b-4190-8456-8a2b965ad3e7" volumeName="kubernetes.io/projected/1b38e492-9c4b-4190-8456-8a2b965ad3e7-kube-api-access-9vgmh" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853209    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d81696-4204-4290-81f7-8cfe42e3bed0" volumeName="kubernetes.io/projected/60d81696-4204-4290-81f7-8cfe42e3bed0-kube-api-access-ddwwx" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853217    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b8cdafe-c2f7-433c-aac4-f4c8c6adf870" volumeName="kubernetes.io/projected/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-kube-api-access-9tgq4" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853225    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283a86d8-9e77-4f36-8fa9-0bd380ec043f" volumeName="kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853231    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9118c60c-2783-4f6e-8f31-2f5bd6f8d684" volumeName="kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-env-overrides" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853240    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a" volumeName="kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853248    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a" volumeName="kubernetes.io/empty-dir/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-data-volume" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853256    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f" volumeName="kubernetes.io/configmap/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-serviceca" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853263    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b38e492-9c4b-4190-8456-8a2b965ad3e7" volumeName="kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cni-binary-copy" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853271    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283a86d8-9e77-4f36-8fa9-0bd380ec043f" volumeName="kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853279    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d81696-4204-4290-81f7-8cfe42e3bed0" volumeName="kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-tuned" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853293    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71142023-8714-4106-8db9-e3522b32da9e" volumeName="kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853321    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12e057a9-159b-465f-9864-3d4521016ac6" volumeName="kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853328    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="178af0c6-6aaf-4a57-9ed4-25988a9bac33" volumeName="kubernetes.io/empty-dir/178af0c6-6aaf-4a57-9ed4-25988a9bac33-tmp-dir" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853335    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="211397c2-9e2b-48fe-9fb4-050a6cba9a5c" volumeName="kubernetes.io/configmap/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-iptables-alerter-script" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853344    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7740f0f0-41ba-4325-9a49-1ff07a8c8475" volumeName="kubernetes.io/projected/7740f0f0-41ba-4325-9a49-1ff07a8c8475-kube-api-access-m6qmr" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853351    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70276149-d6d6-4728-9ffd-9d9351c23ac6" volumeName="kubernetes.io/secret/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-key" seLinuxMountContext=""
Nov 02 21:08:32.853349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853358    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8444a0ef-d9fe-4605-97e3-6d6263228a11" volumeName="kubernetes.io/secret/8444a0ef-d9fe-4605-97e3-6d6263228a11-agent-certs" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853371    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12e057a9-159b-465f-9864-3d4521016ac6" volumeName="kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-binary-copy" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853381    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="178af0c6-6aaf-4a57-9ed4-25988a9bac33" volumeName="kubernetes.io/projected/178af0c6-6aaf-4a57-9ed4-25988a9bac33-kube-api-access-jn6h5" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853388    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="211397c2-9e2b-48fe-9fb4-050a6cba9a5c" volumeName="kubernetes.io/projected/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-kube-api-access-89fr2" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853413    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d81696-4204-4290-81f7-8cfe42e3bed0" volumeName="kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-tmp" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853421    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a" volumeName="kubernetes.io/projected/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-api-access-xwzls" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853430    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283a86d8-9e77-4f36-8fa9-0bd380ec043f" volumeName="kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853439    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70276149-d6d6-4728-9ffd-9d9351c23ac6" volumeName="kubernetes.io/projected/70276149-d6d6-4728-9ffd-9d9351c23ac6-kube-api-access-ps67r" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853454    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9118c60c-2783-4f6e-8f31-2f5bd6f8d684" volumeName="kubernetes.io/secret/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853464    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9941cc1-f486-4fbc-bc93-9b6070db00d3" volumeName="kubernetes.io/projected/a9941cc1-f486-4fbc-bc93-9b6070db00d3-kube-api-access-f2vm8" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853472    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f" volumeName="kubernetes.io/projected/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-kube-api-access-6zs6t" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853404    4528 factory.go:153] Registering CRI-O factory
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853499    4528 factory.go:223] Registration of the crio container factory successfully
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853555    4528 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853575    4528 factory.go:103] Registering Raw factory
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853590    4528 manager.go:1199] Started watching for new ooms in manager
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853479    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283a86d8-9e77-4f36-8fa9-0bd380ec043f" volumeName="kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853615    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70276149-d6d6-4728-9ffd-9d9351c23ac6" volumeName="kubernetes.io/configmap/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-cabundle" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853624    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8444a0ef-d9fe-4605-97e3-6d6263228a11" volumeName="kubernetes.io/configmap/8444a0ef-d9fe-4605-97e3-6d6263228a11-konnectivity-ca" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853631    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8abc6131-3792-4669-bfe7-a1c88ee317a3" volumeName="kubernetes.io/projected/8abc6131-3792-4669-bfe7-a1c88ee317a3-kube-api-access-stph2" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853638    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9118c60c-2783-4f6e-8f31-2f5bd6f8d684" volumeName="kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853646    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9118c60c-2783-4f6e-8f31-2f5bd6f8d684" volumeName="kubernetes.io/projected/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-kube-api-access-2scqv" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853653    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12e057a9-159b-465f-9864-3d4521016ac6" volumeName="kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853660    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12e057a9-159b-465f-9864-3d4521016ac6" volumeName="kubernetes.io/projected/12e057a9-159b-465f-9864-3d4521016ac6-kube-api-access-r9hr2" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853669    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="178af0c6-6aaf-4a57-9ed4-25988a9bac33" volumeName="kubernetes.io/configmap/178af0c6-6aaf-4a57-9ed4-25988a9bac33-config-volume" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853681    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9118c60c-2783-4f6e-8f31-2f5bd6f8d684" volumeName="kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-config" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853692    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9941cc1-f486-4fbc-bc93-9b6070db00d3" volumeName="kubernetes.io/empty-dir/a9941cc1-f486-4fbc-bc93-9b6070db00d3-tmp-dir" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853700    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9be1c6f-dae7-418e-ae7d-d0cd42578469" volumeName="kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret" seLinuxMountContext=""
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853707    4528 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853711    4528 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 21:08:32.854272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.853946    4528 manager.go:319] Starting recovery of all containers
Nov 02 21:08:32.902958 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.902918    4528 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 21:08:32.906491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.905442    4528 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 21:08:32.906491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.905465    4528 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 21:08:32.906491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.905486    4528 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 21:08:32.906491 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:32.905531    4528 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 21:08:32.930631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.930540    4528 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="23abb1e9877c10ffc7f7858e4c4cd60b4c97462e75feba500f408c4fa9e3c0d7" exitCode=0
Nov 02 21:08:32.930781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.930768    4528 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="1b95a9893290c6ca5f91a8a6c17d2f44fff9d11bc0f67619b96a9787b0870edc" exitCode=0
Nov 02 21:08:32.930857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.930847    4528 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="8050bd793e1694f6586d68c913699c42ff7639e31dd9aef359ca234f2833a32e" exitCode=0
Nov 02 21:08:32.930924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.930915    4528 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="c018a0a17a4ff11b78e1c949647347e0cd54e43cf66ae6c22884a93ca6a0eb1c" exitCode=0
Nov 02 21:08:32.930983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.930974    4528 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="fd73cd889359fda6ae3d8ee007ecb247cda4226755e6140d1bf011faa004e500" exitCode=0
Nov 02 21:08:32.931040 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.931031    4528 generic.go:356] "Generic (PLEG): container finished" podID="12e057a9-159b-465f-9864-3d4521016ac6" containerID="387b3467cba5d40b40a104f3e1eb338063c8db50adfebb57da2e391952433c70" exitCode=0
Nov 02 21:08:32.951018 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:32.950994    4528 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:08:32.957502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.957467    4528 generic.go:356] "Generic (PLEG): container finished" podID="65299bdb8e943305caf24f1bcda4b12f" containerID="2f14b63c86de9e3dc11179c211a02ab1232a49ccb04206e55be1db614dfdc606" exitCode=0
Nov 02 21:08:32.980857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.980836    4528 manager.go:324] Recovery completed
Nov 02 21:08:32.999005 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:32.998977    4528 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:08:33.000453 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.000433    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:08:33.000562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.000463    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:08:33.000562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.000475    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:08:33.002056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002040    4528 cpu_manager.go:226] "Starting" policy="none"
Nov 02 21:08:33.002056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002053    4528 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 21:08:33.002207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002060    4528 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:08:33.002207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002071    4528 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 21:08:33.002262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002245    4528 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 21:08:33.002294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002261    4528 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 21:08:33.002294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002283    4528 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 21:08:33.002294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002288    4528 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 21:08:33.002294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002294    4528 policy_none.go:50] "Start"
Nov 02 21:08:33.002407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002302    4528 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 21:08:33.002407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002313    4528 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 21:08:33.002458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002443    4528 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 21:08:33.002458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002450    4528 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 21:08:33.002458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.002455    4528 policy_none.go:44] "Start"
Nov 02 21:08:33.003203 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.003185    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:08:33.003268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.003220    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:08:33.003268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.003234    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:08:33.005600 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:33.005574    4528 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 21:08:33.006614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.006593    4528 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 21:08:33.009251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.009232    4528 manager.go:341] "Starting Device Plugin manager"
Nov 02 21:08:33.009351 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:33.009265    4528 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 21:08:33.009351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.009278    4528 server.go:87] "Starting device plugin registration server"
Nov 02 21:08:33.009609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.009596    4528 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 21:08:33.009678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.009610    4528 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 21:08:33.010475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.009751    4528 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 21:08:33.010475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.009838    4528 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 21:08:33.010475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.009873    4528 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 21:08:33.010475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.009880    4528 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 21:08:33.010712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.009932    4528 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:08:33.009863158Z","UUID":"5b4b9717-bb99-4a5d-8267-5e67f6de37e1","Handler":null,"Name":"","Endpoint":""}
Nov 02 21:08:33.011276 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:33.011252    4528 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 21:08:33.011358 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:33.011299    4528 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-86.us-west-2.compute.internal\" not found"
Nov 02 21:08:33.011896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.011879    4528 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 21:08:33.011965 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.011901    4528 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 21:08:33.035154 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.035121    4528 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:08:33.035325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.035230    4528 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 21:08:33.039823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.039804    4528 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:08:33.040079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.040063    4528 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:08:33.086883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.086847    4528 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 21:08:33.111934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.111898    4528 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 21:08:33.113262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.113240    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 21:08:33.113340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.113277    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 21:08:33.113340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.113291    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 21:08:33.113340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.113328    4528 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.206064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.205947    4528 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"]
Nov 02 21:08:33.206592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.206536    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"65299bdb8e943305caf24f1bcda4b12f","Type":"ContainerStarted","Data":"ff613c5444f1fccade6a761281c12749398a423aa11e5aa5835ccec2590c0ac9"}
Nov 02 21:08:33.206592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.206586    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"65299bdb8e943305caf24f1bcda4b12f","Type":"ContainerDied","Data":"2f14b63c86de9e3dc11179c211a02ab1232a49ccb04206e55be1db614dfdc606"}
Nov 02 21:08:33.206744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.206606    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"65299bdb8e943305caf24f1bcda4b12f","Type":"ContainerStarted","Data":"83cddc74cea102b6aa974c9c5c094dee54bc7b01e929cdfaf219c2cf600343dc"}
Nov 02 21:08:33.206744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.206624    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"32f26124355cba5ec51ff80a5481f572","Type":"ContainerStarted","Data":"c49e25f1faf0dc4e264eb38472058a43b169bbae5ac936e9ce9c96f95cf8a663"}
Nov 02 21:08:33.206744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.206642    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal" event={"ID":"32f26124355cba5ec51ff80a5481f572","Type":"ContainerStarted","Data":"6ec3711a66e817b4bce816161483d39b166c56a5145291e10214e65bfb56f885"}
Nov 02 21:08:33.206744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.206661    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.206937 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.206773    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.215856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.215824    4528 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.215980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.215905    4528 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.263602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.263568    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:08:33.263747 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:33.263631    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.263916 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.263899    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:08:33.263947 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:33.263939    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.337804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.337767    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.337960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.337814    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.337960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.337833    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32f26124355cba5ec51ff80a5481f572-config\") pod \"kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"32f26124355cba5ec51ff80a5481f572\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.438930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.438890    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.439071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.438944    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.439071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.438965    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32f26124355cba5ec51ff80a5481f572-config\") pod \"kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"32f26124355cba5ec51ff80a5481f572\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.439071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.438899    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.439071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.439004    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32f26124355cba5ec51ff80a5481f572-config\") pod \"kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"32f26124355cba5ec51ff80a5481f572\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.439071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.439039    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65299bdb8e943305caf24f1bcda4b12f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" (UID: \"65299bdb8e943305caf24f1bcda4b12f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.836773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.836742    4528 apiserver.go:52] "Watching apiserver"
Nov 02 21:08:33.979430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.979403    4528 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 21:08:33.979960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.979930    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-zc285","openshift-network-diagnostics/network-check-target-plprl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal","openshift-dns/dns-default-wf86r","openshift-insights/insights-runtime-extractor-ctfx4","openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f","kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7","openshift-cluster-node-tuning-operator/tuned-77cx8","openshift-multus/network-metrics-daemon-zf2t8","openshift-ingress-canary/ingress-canary-n64lc","openshift-insights/periodic-gathering-8924h-85j4m","openshift-multus/multus-additional-cni-plugins-rp6h8","openshift-ovn-kubernetes/ovnkube-node-2lfm8","openshift-image-registry/node-ca-sllxw","openshift-network-operator/iptables-alerter-phqtr","openshift-service-ca/service-ca-5fffc97bdf-748zb","openshift-multus/multus-cz9xl","kube-system/global-pull-secret-syncer-zl2hk","kube-system/konnectivity-agent-qzbfv"]
Nov 02 21:08:33.980485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.980470    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.980572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.980553    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:33.980718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.980697    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:33.988309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.988283    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:33.988744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:33.988723    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.031081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031047    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.031081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031055    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 21:08:34.031425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031402    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.031505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031404    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.031505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031440    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.031734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031511    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.031734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031557    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 21:08:34.031734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031562    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.031734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031601    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.031734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031676    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 21:08:34.031734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.031704    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.032134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032103    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.032180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032167    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.032281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032262    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 21:08:34.032341 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032268    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 21:08:34.032341 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032287    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 21:08:34.032341 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032294    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 21:08:34.032341 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032335    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 21:08:34.032526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032435    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 21:08:34.032748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032733    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 21:08:34.032805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032795    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 21:08:34.032805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.032802    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z9mmm\""
Nov 02 21:08:34.033057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033036    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvdw\""
Nov 02 21:08:34.033057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033043    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v96df\""
Nov 02 21:08:34.033249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033079    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 21:08:34.033249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033146    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9rn4m\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033270    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fx9vz\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033296    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033300    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbqzp\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033310    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9vsgv\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033311    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnfwz\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033313    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033338    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-894d2\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033297    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033322    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.033352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033356    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 21:08:34.033645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033372    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 21:08:34.033645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033420    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2mhzd\""
Nov 02 21:08:34.033645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.033464    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8dkx\""
Nov 02 21:08:34.036177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.036161    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.036233 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.036190    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.039437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.039424    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 21:08:34.053815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.053793    4528 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 21:08:34.076302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.076277    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.076401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.076325    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d9psf\""
Nov 02 21:08:34.076401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.076356    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.077680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.077665    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9p4p6\""
Nov 02 21:08:34.077743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.077707    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.077743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.077715    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 21:08:34.077743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.077708    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.077743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.077727    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 21:08:34.078473 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.078459    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:08:34.078523 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.078504    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:34.079350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.079337    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 21:08:34.079413 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.079386    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.us-west-2.compute.internal"
Nov 02 21:08:34.079973 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.079958    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.080054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080013    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 21:08:34.080054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080021    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 21:08:34.080054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080051    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 21:08:34.080214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080058    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 21:08:34.080214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080076    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 21:08:34.080214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080134    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 21:08:34.080214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080140    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jb7zz\""
Nov 02 21:08:34.080214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080163    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.080214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080158    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 21:08:34.080214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080203    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 21:08:34.080214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080204    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9hv44\""
Nov 02 21:08:34.080461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080228    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5jdjp\""
Nov 02 21:08:34.080461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080228    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 21:08:34.080461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080280    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ltkgn\""
Nov 02 21:08:34.080461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.080356    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-58fxm\""
Nov 02 21:08:34.143940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.143866    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-system-cni-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.143940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.143900    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.143940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.143916    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-run\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.143940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.143933    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-serviceca\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:08:34.144303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.143983    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89fr2\" (UniqueName: \"kubernetes.io/projected/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-kube-api-access-89fr2\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:08:34.144303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144021    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-systemd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.144303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144039    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-crio-socket\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.144303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144205    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-kubernetes\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.144303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144229    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/178af0c6-6aaf-4a57-9ed4-25988a9bac33-config-volume\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.144303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144250    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-netns\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.144303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144255    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-serviceca\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:08:34.144303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144284    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-bin\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.144303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144304    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-conf-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.144618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144286    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.144618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144334    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgmh\" (UniqueName: \"kubernetes.io/projected/1b38e492-9c4b-4190-8456-8a2b965ad3e7-kube-api-access-9vgmh\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.144618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144362    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:34.144618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144435    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.144618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144477    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/178af0c6-6aaf-4a57-9ed4-25988a9bac33-config-volume\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.144618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144564    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9be1c6f-dae7-418e-ae7d-d0cd42578469-original-pull-secret\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:34.144618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144568    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-socket-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144622    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-etc-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144650    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144672    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-netd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144698    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-env-overrides\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144756    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cnibin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144782    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-netns\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144809    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144836    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6qmr\" (UniqueName: \"kubernetes.io/projected/7740f0f0-41ba-4325-9a49-1ff07a8c8475-kube-api-access-m6qmr\") pod \"migrator-66df5d496b-lx92f\" (UID: \"7740f0f0-41ba-4325-9a49-1ff07a8c8475\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144864    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-log-socket\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144925    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-env-overrides\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.144959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144941    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-hostroot\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.145317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.144967    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-cnibin\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.145317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145001    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.145317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145025    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-sys-fs\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.145317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145051    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-tuned\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.145317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145076    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwwx\" (UniqueName: \"kubernetes.io/projected/60d81696-4204-4290-81f7-8cfe42e3bed0-kube-api-access-ddwwx\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.145317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145138    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zs6t\" (UniqueName: \"kubernetes.io/projected/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-kube-api-access-6zs6t\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:08:34.145317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145178    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-tuned\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.145317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-var-lib-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.145662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145359    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-ovn\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.145662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145393    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-data-volume\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.145662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145467    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-data-volume\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.145662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145488    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-device-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.145662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145505    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-systemd\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.145662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145521    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-host-slash\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:08:34.145662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145540    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178af0c6-6aaf-4a57-9ed4-25988a9bac33-tmp-dir\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.145662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145556    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-cabundle\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:34.145662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145616    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178af0c6-6aaf-4a57-9ed4-25988a9bac33-tmp-dir\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145682    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovn-node-metrics-cert\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145717    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-system-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145745    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-k8s-cni-cncf-io\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145768    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-cabundle\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145771    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-var-lib-kubelet\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145805    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9941cc1-f486-4fbc-bc93-9b6070db00d3-hosts-file\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145830    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scqv\" (UniqueName: \"kubernetes.io/projected/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-kube-api-access-2scqv\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145853    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-etc-kubernetes\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145865    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovn-node-metrics-cert\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145905    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-dbus\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145943    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.145997    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8444a0ef-d9fe-4605-97e3-6d6263228a11-agent-certs\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:08:34.146089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146038    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-tmp\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146105    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-iptables-alerter-script\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146159    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d81696-4204-4290-81f7-8cfe42e3bed0-tmp\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146168    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146247    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps67r\" (UniqueName: \"kubernetes.io/projected/70276149-d6d6-4728-9ffd-9d9351c23ac6-kube-api-access-ps67r\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146269    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146277    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8444a0ef-d9fe-4605-97e3-6d6263228a11-agent-certs\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146288    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-config\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146322    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146339    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-modprobe-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146347    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-iptables-alerter-script\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146376    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysconfig\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146407    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9941cc1-f486-4fbc-bc93-9b6070db00d3-tmp-dir\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146432    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vm8\" (UniqueName: \"kubernetes.io/projected/a9941cc1-f486-4fbc-bc93-9b6070db00d3-kube-api-access-f2vm8\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146440    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-config\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146455    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-socket-dir-parent\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.146519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146473    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-kubelet\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146567    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-daemon-config\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146584    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146640    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146698    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-conf\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146780    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-sys\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146814    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-host\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146830    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146856    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-binary-copy\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146873    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-daemon-config\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146894    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgq4\" (UniqueName: \"kubernetes.io/projected/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-kube-api-access-9tgq4\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.146929    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-registration-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147019    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njx2v\" (UniqueName: \"kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147061    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147065    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-key\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147102    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147083    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12e057a9-159b-465f-9864-3d4521016ac6-cni-binary-copy\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.147145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147131    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9941cc1-f486-4fbc-bc93-9b6070db00d3-tmp-dir\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147170    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-systemd-units\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147210    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-multus-certs\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147247    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hr2\" (UniqueName: \"kubernetes.io/projected/12e057a9-159b-465f-9864-3d4521016ac6-kube-api-access-r9hr2\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147292    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cni-binary-copy\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147317    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-kubelet-config\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147345    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70276149-d6d6-4728-9ffd-9d9351c23ac6-signing-key\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147347    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8444a0ef-d9fe-4605-97e3-6d6263228a11-konnectivity-ca\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147395    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjs7\" (UniqueName: \"kubernetes.io/projected/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kube-api-access-2pjs7\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147425    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147505    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-bin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147513    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cni-binary-copy\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147538    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzls\" (UniqueName: \"kubernetes.io/projected/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-api-access-xwzls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147548    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8444a0ef-d9fe-4605-97e3-6d6263228a11-konnectivity-ca\") pod \"konnectivity-agent-qzbfv\" (UID: \"8444a0ef-d9fe-4605-97e3-6d6263228a11\") " pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147596    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-cert\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147705    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147735    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stph2\" (UniqueName: \"kubernetes.io/projected/8abc6131-3792-4669-bfe7-a1c88ee317a3-kube-api-access-stph2\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147761    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-slash\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.147819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147796    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-cert\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:34.148503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147844    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:34.148503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.147787    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-multus\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.148503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148021    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:34.148503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148038    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.148503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148057    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-host\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:08:34.148503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148075    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6h5\" (UniqueName: \"kubernetes.io/projected/178af0c6-6aaf-4a57-9ed4-25988a9bac33-kube-api-access-jn6h5\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.148503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148377    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-kubelet\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.148503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148435    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-script-lib\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.148503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148463    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.148877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148526    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:34.148877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:34.148877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148776    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-lib-modules\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.148877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148809    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.148877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.148839    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-node-log\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.149429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.149386    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-ovnkube-script-lib\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.196399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.196365    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgmh\" (UniqueName: \"kubernetes.io/projected/1b38e492-9c4b-4190-8456-8a2b965ad3e7-kube-api-access-9vgmh\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.196583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.196365    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89fr2\" (UniqueName: \"kubernetes.io/projected/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-kube-api-access-89fr2\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:08:34.197025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.197001    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6qmr\" (UniqueName: \"kubernetes.io/projected/7740f0f0-41ba-4325-9a49-1ff07a8c8475-kube-api-access-m6qmr\") pod \"migrator-66df5d496b-lx92f\" (UID: \"7740f0f0-41ba-4325-9a49-1ff07a8c8475\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lx92f"
Nov 02 21:08:34.197137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.197071    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zs6t\" (UniqueName: \"kubernetes.io/projected/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-kube-api-access-6zs6t\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:08:34.197214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.197193    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwwx\" (UniqueName: \"kubernetes.io/projected/60d81696-4204-4290-81f7-8cfe42e3bed0-kube-api-access-ddwwx\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.199961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.199940    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scqv\" (UniqueName: \"kubernetes.io/projected/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-kube-api-access-2scqv\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.200023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.200000    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vm8\" (UniqueName: \"kubernetes.io/projected/a9941cc1-f486-4fbc-bc93-9b6070db00d3-kube-api-access-f2vm8\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:08:34.201327 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.201306    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6h5\" (UniqueName: \"kubernetes.io/projected/178af0c6-6aaf-4a57-9ed4-25988a9bac33-kube-api-access-jn6h5\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.201398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.201344    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzls\" (UniqueName: \"kubernetes.io/projected/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-kube-api-access-xwzls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.201826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.201795    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lmb\" (UniqueName: \"kubernetes.io/projected/71142023-8714-4106-8db9-e3522b32da9e-kube-api-access-t8lmb\") pod \"network-check-target-plprl\" (UID: \"71142023-8714-4106-8db9-e3522b32da9e\") " pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:34.202309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.202288    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stph2\" (UniqueName: \"kubernetes.io/projected/8abc6131-3792-4669-bfe7-a1c88ee317a3-kube-api-access-stph2\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:34.202377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.202352    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njx2v\" (UniqueName: \"kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v\") pod \"periodic-gathering-8924h-85j4m\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") " pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:08:34.202377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.202368    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hr2\" (UniqueName: \"kubernetes.io/projected/12e057a9-159b-465f-9864-3d4521016ac6-kube-api-access-r9hr2\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.202448 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.202356    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjs7\" (UniqueName: \"kubernetes.io/projected/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kube-api-access-2pjs7\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.202448 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.202431    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps67r\" (UniqueName: \"kubernetes.io/projected/70276149-d6d6-4728-9ffd-9d9351c23ac6-kube-api-access-ps67r\") pod \"service-ca-5fffc97bdf-748zb\" (UID: \"70276149-d6d6-4728-9ffd-9d9351c23ac6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-748zb"
Nov 02 21:08:34.202509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.202449    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgq4\" (UniqueName: \"kubernetes.io/projected/0b8cdafe-c2f7-433c-aac4-f4c8c6adf870-kube-api-access-9tgq4\") pod \"ingress-canary-n64lc\" (UID: \"0b8cdafe-c2f7-433c-aac4-f4c8c6adf870\") " pod="openshift-ingress-canary/ingress-canary-n64lc"
Nov 02 21:08:34.250033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.249999    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-slash\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250033    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-multus\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250049    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250066    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-host\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250095    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-kubelet\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250139    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250137    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-slash\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250137    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-multus\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250156    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-lib-modules\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.250225    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250252    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250258    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-lib-modules\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.250271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250260    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250177    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-kubelet\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250240    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.250332    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:34.750308616 +0000 UTC m=+2.013844931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : secret "metrics-daemon-secret" not found
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250175    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f-host\") pod \"node-ca-sllxw\" (UID: \"d7a48f66-64fa-40f0-a8ad-17ffcc05bd2f\") " pod="openshift-image-registry/node-ca-sllxw"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.250346    4528 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250371    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-node-log\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.250399    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls podName:178af0c6-6aaf-4a57-9ed4-25988a9bac33 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:34.750384752 +0000 UTC m=+2.013921064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls") pod "dns-default-wf86r" (UID: "178af0c6-6aaf-4a57-9ed4-25988a9bac33") : secret "dns-default-metrics-tls" not found
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250418    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-system-cni-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250445    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-run\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250461    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-system-cni-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250468    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-systemd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250497    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-crio-socket\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250502    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-systemd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250421    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-node-log\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250521    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-kubernetes\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250547    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-netns\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250555    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-run\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250569    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-bin\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250584    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-crio-socket\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250592    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-netns\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250582    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-kubernetes\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250604    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-conf-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250624    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-bin\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250631    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250655    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-socket-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250664    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-conf-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250677    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-etc-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250705    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250709    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-socket-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250705    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250734    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-netd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250737    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250735    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-etc-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250750    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cnibin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250753    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-cni-netd\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250782    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-netns\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250798    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250813    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-log-socket\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250826    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-hostroot\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250830    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-cnibin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.250815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250839    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-cnibin\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250841    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-netns\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250866    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-log-socket\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250875    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12e057a9-159b-465f-9864-3d4521016ac6-cnibin\") pod \"multus-additional-cni-plugins-rp6h8\" (UID: \"12e057a9-159b-465f-9864-3d4521016ac6\") " pod="openshift-multus/multus-additional-cni-plugins-rp6h8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250883    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-hostroot\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.250900    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.250942    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:34.750927851 +0000 UTC m=+2.014464160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250962    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.250988    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-sys-fs\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-var-lib-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251026    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251039    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-ovn\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251044    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-var-lib-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251046    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-sys-fs\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251064    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-device-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251088    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-systemd\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251133    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-host-slash\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251139    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-device-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251143    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-ovn\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251160    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-system-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251174    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-systemd\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251173    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/211397c2-9e2b-48fe-9fb4-050a6cba9a5c-host-slash\") pod \"iptables-alerter-phqtr\" (UID: \"211397c2-9e2b-48fe-9fb4-050a6cba9a5c\") " pod="openshift-network-operator/iptables-alerter-phqtr"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251184    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-k8s-cni-cncf-io\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251215    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-var-lib-kubelet\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251220    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-k8s-cni-cncf-io\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251218    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-system-cni-dir\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251241    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9941cc1-f486-4fbc-bc93-9b6070db00d3-hosts-file\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251254    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-var-lib-kubelet\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251270    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-etc-kubernetes\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251281    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a9941cc1-f486-4fbc-bc93-9b6070db00d3-hosts-file\") pod \"node-resolver-zc285\" (UID: \"a9941cc1-f486-4fbc-bc93-9b6070db00d3\") " pod="openshift-dns/node-resolver-zc285"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251294    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-dbus\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251322    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-etc-kubernetes\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251324    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251349    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-dbus\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251352    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251376    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251385    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-modprobe-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251352    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251409    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysconfig\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251438    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-socket-dir-parent\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251464    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-modprobe-d\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251462    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-kubelet\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251499    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-kubelet\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251503    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-conf\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251522    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysconfig\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251527    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-multus-socket-dir-parent\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251526    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-sys\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251584    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-host\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251602    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251619    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-etc-sysctl-conf\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251630    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-sys\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251635    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-registration-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251641    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60d81696-4204-4290-81f7-8cfe42e3bed0-host\") pod \"tuned-77cx8\" (UID: \"60d81696-4204-4290-81f7-8cfe42e3bed0\") " pod="openshift-cluster-node-tuning-operator/tuned-77cx8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251665    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c11c4c6b-5bf0-4203-b5c6-2287c0ff122c-registration-dir\") pod \"aws-ebs-csi-driver-node-ntrs7\" (UID: \"c11c4c6b-5bf0-4203-b5c6-2287c0ff122c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntrs7"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251666    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-systemd-units\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251636    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-run-openvswitch\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251694    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-multus-certs\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251716    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9118c60c-2783-4f6e-8f31-2f5bd6f8d684-systemd-units\") pod \"ovnkube-node-2lfm8\" (UID: \"9118c60c-2783-4f6e-8f31-2f5bd6f8d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251726    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-run-multus-certs\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251726    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-kubelet-config\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251751    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-bin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251768    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9be1c6f-dae7-418e-ae7d-d0cd42578469-kubelet-config\") pod \"global-pull-secret-syncer-zl2hk\" (UID: \"b9be1c6f-dae7-418e-ae7d-d0cd42578469\") " pod="kube-system/global-pull-secret-syncer-zl2hk"
Nov 02 21:08:34.252311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.251813    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b38e492-9c4b-4190-8456-8a2b965ad3e7-host-var-lib-cni-bin\") pod \"multus-cz9xl\" (UID: \"1b38e492-9c4b-4190-8456-8a2b965ad3e7\") " pod="openshift-multus/multus-cz9xl"
Nov 02 21:08:34.756859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.756822    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:34.757033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.756866    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:34.757033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:34.756928    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:34.757033 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.756997    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:08:34.757033 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.757012    4528 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:08:34.757257 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.757053    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:34.757257 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.757079    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:35.757063294 +0000 UTC m=+3.020599594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : secret "metrics-daemon-secret" not found
Nov 02 21:08:34.757257 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.757093    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls podName:178af0c6-6aaf-4a57-9ed4-25988a9bac33 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:35.757087446 +0000 UTC m=+3.020623745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls") pod "dns-default-wf86r" (UID: "178af0c6-6aaf-4a57-9ed4-25988a9bac33") : secret "dns-default-metrics-tls" not found
Nov 02 21:08:34.757257 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:34.757140    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:35.757097782 +0000 UTC m=+3.020634089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:35.765191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:35.765146    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:35.765664 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:35.765200    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:35.765664 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:35.765240    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:35.765664 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:35.765297    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:08:35.765664 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:35.765361    4528 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 21:08:35.765664 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:35.765374    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:37.765359789 +0000 UTC m=+5.028896097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : secret "metrics-daemon-secret" not found
Nov 02 21:08:35.765664 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:35.765426    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls podName:178af0c6-6aaf-4a57-9ed4-25988a9bac33 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:37.765409999 +0000 UTC m=+5.028946294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls") pod "dns-default-wf86r" (UID: "178af0c6-6aaf-4a57-9ed4-25988a9bac33") : secret "dns-default-metrics-tls" not found
Nov 02 21:08:35.765664 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:35.765368    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:35.765664 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:35.765467    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:37.765459763 +0000 UTC m=+5.028996058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:37.778955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:37.778923    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:37.778955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:37.778960    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:37.779496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:37.778995    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:37.779496 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:37.779076    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:37.779496 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:37.779150    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:41.779137824 +0000 UTC m=+9.042674132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:37.779496 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:37.779076    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:08:37.779496 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:37.779216    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:41.779197122 +0000 UTC m=+9.042733432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : secret "metrics-daemon-secret" not found
Nov 02 21:08:37.779496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:37.779342    4528 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 21:08:37.781304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:37.781288    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/178af0c6-6aaf-4a57-9ed4-25988a9bac33-metrics-tls\") pod \"dns-default-wf86r\" (UID: \"178af0c6-6aaf-4a57-9ed4-25988a9bac33\") " pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:37.889413 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:37.889373    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:38.250358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:38.250289    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wf86r"]
Nov 02 21:08:38.253908 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:38.253860    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178af0c6_6aaf_4a57_9ed4_25988a9bac33.slice/crio-97d4350ebbe66695c0dd8646f3b92b3fd69d1a0c4adddf30165030d648200eb9 WatchSource:0}: Error finding container 97d4350ebbe66695c0dd8646f3b92b3fd69d1a0c4adddf30165030d648200eb9: Status 404 returned error can't find the container with id 97d4350ebbe66695c0dd8646f3b92b3fd69d1a0c4adddf30165030d648200eb9
Nov 02 21:08:38.255980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:38.255965    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:08:38.756194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:38.756165    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qzbfv"
Nov 02 21:08:38.994876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:38.994840    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wf86r" event={"ID":"178af0c6-6aaf-4a57-9ed4-25988a9bac33","Type":"ContainerStarted","Data":"97d4350ebbe66695c0dd8646f3b92b3fd69d1a0c4adddf30165030d648200eb9"}
Nov 02 21:08:40.123184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:40.123147    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:08:40.227992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:40.227960    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:40.528572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:40.528546    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:41.003400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:41.003361    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wf86r" event={"ID":"178af0c6-6aaf-4a57-9ed4-25988a9bac33","Type":"ContainerStarted","Data":"5f98e06ed019e8e7bd667be3cb4d9ecb6ffaf517f86175cc2b4cbcab9a7ab3b8"}
Nov 02 21:08:41.003580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:41.003422    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wf86r" event={"ID":"178af0c6-6aaf-4a57-9ed4-25988a9bac33","Type":"ContainerStarted","Data":"c7a8ed91b1a99338d58ed52ab863de1f70b6255f0fe65d4b676a14347ddae2da"}
Nov 02 21:08:41.003580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:41.003506    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:41.003580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:41.003537    4528 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 21:08:41.809073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:41.809014    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:41.809560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:41.809097    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:41.809560 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:41.809190    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 21:08:41.809560 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:41.809220    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:41.809560 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:41.809256    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs podName:8abc6131-3792-4669-bfe7-a1c88ee317a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:49.809241635 +0000 UTC m=+17.072777947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs") pod "network-metrics-daemon-zf2t8" (UID: "8abc6131-3792-4669-bfe7-a1c88ee317a3") : secret "metrics-daemon-secret" not found
Nov 02 21:08:41.809560 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:08:41.809291    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls podName:b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a nodeName:}" failed. No retries permitted until 2026-11-02 21:08:49.809274361 +0000 UTC m=+17.072810671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctfx4" (UID: "b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a") : secret "insights-runtime-extractor-tls" not found
Nov 02 21:08:46.008778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.008737    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"]
Nov 02 21:08:46.011701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.011680    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"
Nov 02 21:08:46.041446 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.041407    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8fbd01c-b980-4c76-825d-acae667c2904-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zj6gn\" (UID: \"f8fbd01c-b980-4c76-825d-acae667c2904\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"
Nov 02 21:08:46.099376 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.099338    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-t5gp6\""
Nov 02 21:08:46.104380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.104352    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 21:08:46.111990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.111962    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"]
Nov 02 21:08:46.112442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.112422    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f76f7ddd8-g27bz"]
Nov 02 21:08:46.115581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.115558    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.141859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.141824    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8fbd01c-b980-4c76-825d-acae667c2904-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zj6gn\" (UID: \"f8fbd01c-b980-4c76-825d-acae667c2904\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"
Nov 02 21:08:46.141859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.141864    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c469595b-02ba-41aa-8681-7adf27d5e0cd-ca-trust-extracted\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.142070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.141885    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c469595b-02ba-41aa-8681-7adf27d5e0cd-trusted-ca\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.142070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.141915    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c469595b-02ba-41aa-8681-7adf27d5e0cd-installation-pull-secrets\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.142070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.141961    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c469595b-02ba-41aa-8681-7adf27d5e0cd-bound-sa-token\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.142070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.142012    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c469595b-02ba-41aa-8681-7adf27d5e0cd-registry-tls\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.142070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.142060    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj9gn\" (UniqueName: \"kubernetes.io/projected/c469595b-02ba-41aa-8681-7adf27d5e0cd-kube-api-access-bj9gn\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.142286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.142251    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c469595b-02ba-41aa-8681-7adf27d5e0cd-registry-certificates\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.142327 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.142291    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c469595b-02ba-41aa-8681-7adf27d5e0cd-image-registry-private-configuration\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.144501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.144476    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8fbd01c-b980-4c76-825d-acae667c2904-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zj6gn\" (UID: \"f8fbd01c-b980-4c76-825d-acae667c2904\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"
Nov 02 21:08:46.187081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.187054    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lfm8"
Nov 02 21:08:46.213894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.213863    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 21:08:46.214063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.213914    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 21:08:46.216088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.216068    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qshj9\""
Nov 02 21:08:46.216213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.216148    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 21:08:46.216213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.216178    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 21:08:46.217328 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.217308    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f76f7ddd8-g27bz"]
Nov 02 21:08:46.243191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.243158    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c469595b-02ba-41aa-8681-7adf27d5e0cd-ca-trust-extracted\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.243191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.243190    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c469595b-02ba-41aa-8681-7adf27d5e0cd-trusted-ca\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.243433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.243254    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c469595b-02ba-41aa-8681-7adf27d5e0cd-installation-pull-secrets\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.243433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.243277    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c469595b-02ba-41aa-8681-7adf27d5e0cd-bound-sa-token\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.243433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.243309    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c469595b-02ba-41aa-8681-7adf27d5e0cd-registry-tls\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.243433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.243335    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj9gn\" (UniqueName: \"kubernetes.io/projected/c469595b-02ba-41aa-8681-7adf27d5e0cd-kube-api-access-bj9gn\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.243433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.243388    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c469595b-02ba-41aa-8681-7adf27d5e0cd-registry-certificates\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.243433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.243418    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c469595b-02ba-41aa-8681-7adf27d5e0cd-image-registry-private-configuration\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.243714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.243654    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c469595b-02ba-41aa-8681-7adf27d5e0cd-ca-trust-extracted\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.244361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.244329    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c469595b-02ba-41aa-8681-7adf27d5e0cd-registry-certificates\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.244494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.244480    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c469595b-02ba-41aa-8681-7adf27d5e0cd-trusted-ca\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.245913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.245882    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c469595b-02ba-41aa-8681-7adf27d5e0cd-image-registry-private-configuration\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.246193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.246164    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c469595b-02ba-41aa-8681-7adf27d5e0cd-registry-tls\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.246252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.246213    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c469595b-02ba-41aa-8681-7adf27d5e0cd-installation-pull-secrets\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.296804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.296765    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c469595b-02ba-41aa-8681-7adf27d5e0cd-bound-sa-token\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.297564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.297538    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj9gn\" (UniqueName: \"kubernetes.io/projected/c469595b-02ba-41aa-8681-7adf27d5e0cd-kube-api-access-bj9gn\") pod \"image-registry-f76f7ddd8-g27bz\" (UID: \"c469595b-02ba-41aa-8681-7adf27d5e0cd\") " pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.324760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.324720    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"
Nov 02 21:08:46.428010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.427971    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:46.685747 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.685618    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"]
Nov 02 21:08:46.690030 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:46.689997    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fbd01c_b980_4c76_825d_acae667c2904.slice/crio-1ccc2bca56fee869ce468d6e5b41a2dd96332a1f651b5acefb94882e092b3829 WatchSource:0}: Error finding container 1ccc2bca56fee869ce468d6e5b41a2dd96332a1f651b5acefb94882e092b3829: Status 404 returned error can't find the container with id 1ccc2bca56fee869ce468d6e5b41a2dd96332a1f651b5acefb94882e092b3829
Nov 02 21:08:46.791897 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:46.791862    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f76f7ddd8-g27bz"]
Nov 02 21:08:46.795197 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:46.795167    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc469595b_02ba_41aa_8681_7adf27d5e0cd.slice/crio-5ca7df831b51e0b595f7c9b3baf922e81184ad8edc9257acaf60d64d86b6f383 WatchSource:0}: Error finding container 5ca7df831b51e0b595f7c9b3baf922e81184ad8edc9257acaf60d64d86b6f383: Status 404 returned error can't find the container with id 5ca7df831b51e0b595f7c9b3baf922e81184ad8edc9257acaf60d64d86b6f383
Nov 02 21:08:47.025838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:47.025799    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz" event={"ID":"c469595b-02ba-41aa-8681-7adf27d5e0cd","Type":"ContainerStarted","Data":"395f2f135b4b854dc7e62fa35600039de8e4c7ac3b597eb7b175c2368656bd81"}
Nov 02 21:08:47.026334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:47.025860    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz" event={"ID":"c469595b-02ba-41aa-8681-7adf27d5e0cd","Type":"ContainerStarted","Data":"5ca7df831b51e0b595f7c9b3baf922e81184ad8edc9257acaf60d64d86b6f383"}
Nov 02 21:08:47.026334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:47.025884    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:08:47.027233 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:47.027209    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn" event={"ID":"f8fbd01c-b980-4c76-825d-acae667c2904","Type":"ContainerStarted","Data":"1ccc2bca56fee869ce468d6e5b41a2dd96332a1f651b5acefb94882e092b3829"}
Nov 02 21:08:47.135252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:47.135177    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz" podStartSLOduration=1.135156868 podStartE2EDuration="1.135156868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:47.134281245 +0000 UTC m=+14.397817554" watchObservedRunningTime="2026-11-02 21:08:47.135156868 +0000 UTC m=+14.398693269"
Nov 02 21:08:49.035352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.035308    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn" event={"ID":"f8fbd01c-b980-4c76-825d-acae667c2904","Type":"ContainerStarted","Data":"a0d10ff9c0d78ca54d9860ebd6356cf8932cc8bcfa463a81ac52a64dcd80fa70"}
Nov 02 21:08:49.035821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.035504    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"
Nov 02 21:08:49.039950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.039928    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn"
Nov 02 21:08:49.141769 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.141716    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zj6gn" podStartSLOduration=2.318318496 podStartE2EDuration="4.1417033s" totalImagesPullingTime="1.823384804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:49.141307214 +0000 UTC m=+16.404843533" watchObservedRunningTime="2026-11-02 21:08:49.1417033 +0000 UTC m=+16.405239611"
Nov 02 21:08:49.874337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.874295    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:49.874337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.874348    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:49.876671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.876646    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8abc6131-3792-4669-bfe7-a1c88ee317a3-metrics-certs\") pod \"network-metrics-daemon-zf2t8\" (UID: \"8abc6131-3792-4669-bfe7-a1c88ee317a3\") " pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:49.876777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.876673    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctfx4\" (UID: \"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a\") " pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:49.881605 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.881582    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf2t8"
Nov 02 21:08:49.888795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:49.888773    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ctfx4"
Nov 02 21:08:50.246366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.246279    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zf2t8"]
Nov 02 21:08:50.286753 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:50.286723    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abc6131_3792_4669_bfe7_a1c88ee317a3.slice/crio-770048d1521f1f83b5fc46ef2b38486e85bcc1210554c6e38df6783f9bd0f23e WatchSource:0}: Error finding container 770048d1521f1f83b5fc46ef2b38486e85bcc1210554c6e38df6783f9bd0f23e: Status 404 returned error can't find the container with id 770048d1521f1f83b5fc46ef2b38486e85bcc1210554c6e38df6783f9bd0f23e
Nov 02 21:08:50.295604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.295580    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ctfx4"]
Nov 02 21:08:50.298562 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:50.298535    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b2c6cf_3efd_45da_a0d9_f2ee73c5365a.slice/crio-a2e8b683a27ae8e4c573f920a3970dd78e4fbef30175b2c401664cf39368ceb1 WatchSource:0}: Error finding container a2e8b683a27ae8e4c573f920a3970dd78e4fbef30175b2c401664cf39368ceb1: Status 404 returned error can't find the container with id a2e8b683a27ae8e4c573f920a3970dd78e4fbef30175b2c401664cf39368ceb1
Nov 02 21:08:50.484475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.484438    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"]
Nov 02 21:08:50.487673 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.487651    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.537162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.537137    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2w6gq\""
Nov 02 21:08:50.537162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.537149    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 21:08:50.537375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.537210    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 21:08:50.537597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.537580    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 21:08:50.537654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.537615    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 21:08:50.580088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.580055    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5cm9\" (UniqueName: \"kubernetes.io/projected/82467e24-4ccf-403d-915d-8b99419e3830-kube-api-access-x5cm9\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.580267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.580093    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82467e24-4ccf-403d-915d-8b99419e3830-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.580267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.580127    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82467e24-4ccf-403d-915d-8b99419e3830-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.580267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.580259    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82467e24-4ccf-403d-915d-8b99419e3830-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.583603 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.583582    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 21:08:50.584336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.584318    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"]
Nov 02 21:08:50.680983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.680929    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82467e24-4ccf-403d-915d-8b99419e3830-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.680983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.680984    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5cm9\" (UniqueName: \"kubernetes.io/projected/82467e24-4ccf-403d-915d-8b99419e3830-kube-api-access-x5cm9\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.681232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.681011    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82467e24-4ccf-403d-915d-8b99419e3830-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.681232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.681030    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82467e24-4ccf-403d-915d-8b99419e3830-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.681830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.681810    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82467e24-4ccf-403d-915d-8b99419e3830-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.683385 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.683367    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82467e24-4ccf-403d-915d-8b99419e3830-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.683479 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.683462    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82467e24-4ccf-403d-915d-8b99419e3830-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.734851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.734823    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5cm9\" (UniqueName: \"kubernetes.io/projected/82467e24-4ccf-403d-915d-8b99419e3830-kube-api-access-x5cm9\") pod \"prometheus-operator-9cd89b74c-rk9ql\" (UID: \"82467e24-4ccf-403d-915d-8b99419e3830\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:50.801098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:50.801003    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"
Nov 02 21:08:51.009162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:51.009129    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wf86r"
Nov 02 21:08:51.042243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:51.042202    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctfx4" event={"ID":"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a","Type":"ContainerStarted","Data":"915ee9d7b32e486c581ed65b7e12bc8b790597b79b9028fbeef6132d8a830b59"}
Nov 02 21:08:51.042429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:51.042272    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctfx4" event={"ID":"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a","Type":"ContainerStarted","Data":"a2e8b683a27ae8e4c573f920a3970dd78e4fbef30175b2c401664cf39368ceb1"}
Nov 02 21:08:51.045488 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:51.045452    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zf2t8" event={"ID":"8abc6131-3792-4669-bfe7-a1c88ee317a3","Type":"ContainerStarted","Data":"770048d1521f1f83b5fc46ef2b38486e85bcc1210554c6e38df6783f9bd0f23e"}
Nov 02 21:08:51.160336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:51.160255    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql"]
Nov 02 21:08:51.201315 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:51.201285    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82467e24_4ccf_403d_915d_8b99419e3830.slice/crio-b4fba3b4f74a94f9fb6aa096e6d435f5137297fb65288eb6e14cee436d57fd51 WatchSource:0}: Error finding container b4fba3b4f74a94f9fb6aa096e6d435f5137297fb65288eb6e14cee436d57fd51: Status 404 returned error can't find the container with id b4fba3b4f74a94f9fb6aa096e6d435f5137297fb65288eb6e14cee436d57fd51
Nov 02 21:08:52.049856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:52.049755    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql" event={"ID":"82467e24-4ccf-403d-915d-8b99419e3830","Type":"ContainerStarted","Data":"b4fba3b4f74a94f9fb6aa096e6d435f5137297fb65288eb6e14cee436d57fd51"}
Nov 02 21:08:52.051553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:52.051525    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctfx4" event={"ID":"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a","Type":"ContainerStarted","Data":"450334497cb87a96f22879bb6889d40344e2dadc112921acb7e755ab6592a918"}
Nov 02 21:08:52.053157 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:52.053135    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zf2t8" event={"ID":"8abc6131-3792-4669-bfe7-a1c88ee317a3","Type":"ContainerStarted","Data":"76e73b9516c4553ac3d3a613905babb185eccbf9adece15ef12cbf364a867f1d"}
Nov 02 21:08:52.053261 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:52.053175    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zf2t8" event={"ID":"8abc6131-3792-4669-bfe7-a1c88ee317a3","Type":"ContainerStarted","Data":"59eb7cf96aa863999e6dbf2ed58e1afd294caddbedde47fa7149082a0949bbc0"}
Nov 02 21:08:54.062007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:54.061971    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctfx4" event={"ID":"b6b2c6cf-3efd-45da-a0d9-f2ee73c5365a","Type":"ContainerStarted","Data":"d12981c315be8929038a2c6d2f41ca59a79622e4bcc30e1b526b4e529ae5e29f"}
Nov 02 21:08:54.063763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:54.063738    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql" event={"ID":"82467e24-4ccf-403d-915d-8b99419e3830","Type":"ContainerStarted","Data":"10db746f398f32bbb39080f60b69609ac38b939219bb00a599bb0f6ccf79964c"}
Nov 02 21:08:54.063890 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:54.063786    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql" event={"ID":"82467e24-4ccf-403d-915d-8b99419e3830","Type":"ContainerStarted","Data":"6294a7222032b25e37dbd8d57eecffad6f95c2c4999199bde692905f28059d6e"}
Nov 02 21:08:54.277779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:54.277720    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rk9ql" podStartSLOduration=2.426999162 podStartE2EDuration="4.277705158s" totalImagesPullingTime="1.850705996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:54.277286043 +0000 UTC m=+21.540822360" watchObservedRunningTime="2026-11-02 21:08:54.277705158 +0000 UTC m=+21.541241476"
Nov 02 21:08:57.153157 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.153118    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-swfcc"]
Nov 02 21:08:57.156671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.156650    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.208586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.208558    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 21:08:57.230066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.230029    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13c612c3-8753-423a-b670-9ed1ccb628e6-root\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.230226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.230075    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13c612c3-8753-423a-b670-9ed1ccb628e6-sys\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.230226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.230097    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-tls\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.230324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.230246    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-427kw\" (UniqueName: \"kubernetes.io/projected/13c612c3-8753-423a-b670-9ed1ccb628e6-kube-api-access-427kw\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.230324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.230276    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.230324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.230294    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-textfile\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.230324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.230311    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c612c3-8753-423a-b670-9ed1ccb628e6-metrics-client-ca\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.230440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.230396    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-wtmp\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.230440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.230426    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.252603 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.252578    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b7xvl\""
Nov 02 21:08:57.252662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.252623    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 21:08:57.252662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.252656    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 21:08:57.331600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331569    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-wtmp\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331606    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331636    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13c612c3-8753-423a-b670-9ed1ccb628e6-root\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331688    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13c612c3-8753-423a-b670-9ed1ccb628e6-root\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331715    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13c612c3-8753-423a-b670-9ed1ccb628e6-sys\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331733    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-tls\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331756    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-427kw\" (UniqueName: \"kubernetes.io/projected/13c612c3-8753-423a-b670-9ed1ccb628e6-kube-api-access-427kw\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331770    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-wtmp\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331774    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331818    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-textfile\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331824    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13c612c3-8753-423a-b670-9ed1ccb628e6-sys\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.331911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.331843    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c612c3-8753-423a-b670-9ed1ccb628e6-metrics-client-ca\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.332211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.332184    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-textfile\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.332305 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.332287    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.332395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.332376    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c612c3-8753-423a-b670-9ed1ccb628e6-metrics-client-ca\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.334105 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.334080    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.334242 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.334227    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13c612c3-8753-423a-b670-9ed1ccb628e6-node-exporter-tls\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.385521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.385489    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-427kw\" (UniqueName: \"kubernetes.io/projected/13c612c3-8753-423a-b670-9ed1ccb628e6-kube-api-access-427kw\") pod \"node-exporter-swfcc\" (UID: \"13c612c3-8753-423a-b670-9ed1ccb628e6\") " pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.472808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:57.472725    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-swfcc"
Nov 02 21:08:57.484043 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:08:57.484013    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c612c3_8753_423a_b670_9ed1ccb628e6.slice/crio-5db33d2f8b9ad5dd2b540af6b8998219c97b202a37494d37fa2811c95b105a13 WatchSource:0}: Error finding container 5db33d2f8b9ad5dd2b540af6b8998219c97b202a37494d37fa2811c95b105a13: Status 404 returned error can't find the container with id 5db33d2f8b9ad5dd2b540af6b8998219c97b202a37494d37fa2811c95b105a13
Nov 02 21:08:58.080348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:08:58.080313    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swfcc" event={"ID":"13c612c3-8753-423a-b670-9ed1ccb628e6","Type":"ContainerStarted","Data":"5db33d2f8b9ad5dd2b540af6b8998219c97b202a37494d37fa2811c95b105a13"}
Nov 02 21:09:00.088394 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:00.088366    4528 generic.go:356] "Generic (PLEG): container finished" podID="13c612c3-8753-423a-b670-9ed1ccb628e6" containerID="5303ea725157c4a92848ce9be3753c6e22c848c56385555b6309ec922435f48b" exitCode=0
Nov 02 21:09:00.088764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:00.088409    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swfcc" event={"ID":"13c612c3-8753-423a-b670-9ed1ccb628e6","Type":"ContainerDied","Data":"5303ea725157c4a92848ce9be3753c6e22c848c56385555b6309ec922435f48b"}
Nov 02 21:09:01.093644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:01.093607    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swfcc" event={"ID":"13c612c3-8753-423a-b670-9ed1ccb628e6","Type":"ContainerStarted","Data":"53b3e216fa334fe8ff8f7bd752de8255c3a257594eb645bb1ff794a3608fda09"}
Nov 02 21:09:01.094051 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:01.093670    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swfcc" event={"ID":"13c612c3-8753-423a-b670-9ed1ccb628e6","Type":"ContainerStarted","Data":"d312e1308f3bef78808d5863211b95b8e4a930aa686404aad9c0bb74d630dc5e"}
Nov 02 21:09:01.236598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:01.236547    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-swfcc" podStartSLOduration=1.761594032 podStartE2EDuration="4.236531428s" totalImagesPullingTime="2.396308735s" totalInitContainerRuntime="78.628661ms" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:01.202297898 +0000 UTC m=+28.465834228" watchObservedRunningTime="2026-11-02 21:09:01.236531428 +0000 UTC m=+28.500067744"
Nov 02 21:09:06.531381 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.531347    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-wktvh"]
Nov 02 21:09:06.535288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.535258    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:06.585780 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.585752    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 21:09:06.585962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.585879    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 21:09:06.585962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.585885    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wszsg\""
Nov 02 21:09:06.600306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.600273    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafaae0a-b2a1-487f-968c-4874767897eb-tmp\") pod \"downloads-6dd789c8b7-wktvh\" (UID: \"fafaae0a-b2a1-487f-968c-4874767897eb\") " pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:06.600465 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.600340    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkw6n\" (UniqueName: \"kubernetes.io/projected/fafaae0a-b2a1-487f-968c-4874767897eb-kube-api-access-lkw6n\") pod \"downloads-6dd789c8b7-wktvh\" (UID: \"fafaae0a-b2a1-487f-968c-4874767897eb\") " pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:06.632864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.632826    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-wktvh"]
Nov 02 21:09:06.700945 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.700913    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafaae0a-b2a1-487f-968c-4874767897eb-tmp\") pod \"downloads-6dd789c8b7-wktvh\" (UID: \"fafaae0a-b2a1-487f-968c-4874767897eb\") " pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:06.701160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.700970    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkw6n\" (UniqueName: \"kubernetes.io/projected/fafaae0a-b2a1-487f-968c-4874767897eb-kube-api-access-lkw6n\") pod \"downloads-6dd789c8b7-wktvh\" (UID: \"fafaae0a-b2a1-487f-968c-4874767897eb\") " pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:06.701363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.701339    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafaae0a-b2a1-487f-968c-4874767897eb-tmp\") pod \"downloads-6dd789c8b7-wktvh\" (UID: \"fafaae0a-b2a1-487f-968c-4874767897eb\") " pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:06.754236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.754207    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkw6n\" (UniqueName: \"kubernetes.io/projected/fafaae0a-b2a1-487f-968c-4874767897eb-kube-api-access-lkw6n\") pod \"downloads-6dd789c8b7-wktvh\" (UID: \"fafaae0a-b2a1-487f-968c-4874767897eb\") " pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:06.848600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:06.848520    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:07.316623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.316591    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-wktvh"]
Nov 02 21:09:07.358929 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:09:07.358896    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafaae0a_b2a1_487f_968c_4874767897eb.slice/crio-35adbc5d07ec975b4dc83a8a59d3f4b2ae91b8637dcd0f9db4ad9cf21a397eee WatchSource:0}: Error finding container 35adbc5d07ec975b4dc83a8a59d3f4b2ae91b8637dcd0f9db4ad9cf21a397eee: Status 404 returned error can't find the container with id 35adbc5d07ec975b4dc83a8a59d3f4b2ae91b8637dcd0f9db4ad9cf21a397eee
Nov 02 21:09:07.878267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.878214    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b498f8d68-nhj8d"]
Nov 02 21:09:07.883368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.882349    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:07.980989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.980913    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 21:09:07.980989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.980957    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 21:09:07.980989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.980957    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 21:09:07.981349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.981036    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 21:09:07.981349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.981045    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wcntl\""
Nov 02 21:09:07.981349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.981148    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 21:09:07.981876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:07.981857    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b498f8d68-nhj8d"]
Nov 02 21:09:08.012227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.012194    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-oauth-config\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.012227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.012231    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-config\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.012421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.012251    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-tmp\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.012421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.012269    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-oauth-serving-cert\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.012421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.012394    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-service-ca\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.012519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.012454    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfcmc\" (UniqueName: \"kubernetes.io/projected/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-kube-api-access-pfcmc\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.012519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.012485    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-serving-cert\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.035552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.035525    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f76f7ddd8-g27bz"
Nov 02 21:09:08.113865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.113830    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-oauth-config\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.114083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.113873    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-config\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.114083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.113895    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-tmp\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.114083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.113918    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-oauth-serving-cert\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.114083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.113948    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-service-ca\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.114377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.114319    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcmc\" (UniqueName: \"kubernetes.io/projected/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-kube-api-access-pfcmc\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.114440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.114419    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-serving-cert\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.114830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.114803    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-service-ca\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.114830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.114815    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-config\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.114983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.114860    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-oauth-serving-cert\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.115081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.115063    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-tmp\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.117381 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.117356    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-serving-cert\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.117498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.117387    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-oauth-config\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.118165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.118144    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-wktvh" event={"ID":"fafaae0a-b2a1-487f-968c-4874767897eb","Type":"ContainerStarted","Data":"35adbc5d07ec975b4dc83a8a59d3f4b2ae91b8637dcd0f9db4ad9cf21a397eee"}
Nov 02 21:09:08.169079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.168994    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfcmc\" (UniqueName: \"kubernetes.io/projected/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-kube-api-access-pfcmc\") pod \"console-7b498f8d68-nhj8d\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") " pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.196780 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.196742    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:08.560614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:08.560584    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b498f8d68-nhj8d"]
Nov 02 21:09:08.563926 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:09:08.563896    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f90cdc_dc3c_4ab7_b5d4_34f03992a48e.slice/crio-741410a186a62634ec529fd075b385c5e8505ef6c1d108942bf549ada860d630 WatchSource:0}: Error finding container 741410a186a62634ec529fd075b385c5e8505ef6c1d108942bf549ada860d630: Status 404 returned error can't find the container with id 741410a186a62634ec529fd075b385c5e8505ef6c1d108942bf549ada860d630
Nov 02 21:09:09.122677 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:09.122642    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b498f8d68-nhj8d" event={"ID":"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e","Type":"ContainerStarted","Data":"741410a186a62634ec529fd075b385c5e8505ef6c1d108942bf549ada860d630"}
Nov 02 21:09:10.125931 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:10.125899    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-plprl"
Nov 02 21:09:12.135824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:12.135780    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b498f8d68-nhj8d" event={"ID":"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e","Type":"ContainerStarted","Data":"dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0"}
Nov 02 21:09:12.243379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:12.243233    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b498f8d68-nhj8d" podStartSLOduration=2.114351702 podStartE2EDuration="5.243214943s" totalImagesPullingTime="3.128863241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:12.242950767 +0000 UTC m=+39.506487076" watchObservedRunningTime="2026-11-02 21:09:12.243214943 +0000 UTC m=+39.506751261"
Nov 02 21:09:14.117182 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:09:14.117075    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:09:18.197259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:18.197216    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:18.197853 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:18.197383    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:18.202175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:18.202154    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:19.171436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:19.171403    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:21.815876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.815843    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c95c57675-l5zgg"]
Nov 02 21:09:21.819215 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.819196    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:21.934808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.934768    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-console-config\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:21.934983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.934839    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-oauth-config\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:21.934983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.934955    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-trusted-ca-bundle\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:21.934983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.934982    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-serving-cert\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:21.935099 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.935047    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9h5z\" (UniqueName: \"kubernetes.io/projected/15c573c4-d45e-4414-96c2-0a899e3df524-kube-api-access-p9h5z\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:21.935099 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.935081    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c573c4-d45e-4414-96c2-0a899e3df524-tmp\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:21.935202 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.935173    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-oauth-serving-cert\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:21.935241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:21.935216    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-service-ca\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.007816 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.007756    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 21:09:22.009551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.009526    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c95c57675-l5zgg"]
Nov 02 21:09:22.036070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036040    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-console-config\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.036299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036080    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-oauth-config\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.036299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036101    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-trusted-ca-bundle\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.036299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036138    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-serving-cert\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.036299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036164    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9h5z\" (UniqueName: \"kubernetes.io/projected/15c573c4-d45e-4414-96c2-0a899e3df524-kube-api-access-p9h5z\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.036299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036191    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c573c4-d45e-4414-96c2-0a899e3df524-tmp\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.036299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036219    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-oauth-serving-cert\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.036299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036262    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-service-ca\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.036680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036651    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c573c4-d45e-4414-96c2-0a899e3df524-tmp\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.036977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.036957    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-console-config\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.037129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.037092    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-service-ca\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.037217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.037174    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-trusted-ca-bundle\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.037217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.037202    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-oauth-serving-cert\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.038849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.038830    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-oauth-config\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.039372 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.039352    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-serving-cert\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.090160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.090085    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9h5z\" (UniqueName: \"kubernetes.io/projected/15c573c4-d45e-4414-96c2-0a899e3df524-kube-api-access-p9h5z\") pod \"console-7c95c57675-l5zgg\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") " pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.131854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.131818    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:22.499823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:22.499743    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c95c57675-l5zgg"]
Nov 02 21:09:22.531174 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:09:22.531134    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c573c4_d45e_4414_96c2_0a899e3df524.slice/crio-5fb2d706bc4a36a5dba45cc00ccb612938ceff742c167eaa72b1ab10256c7913 WatchSource:0}: Error finding container 5fb2d706bc4a36a5dba45cc00ccb612938ceff742c167eaa72b1ab10256c7913: Status 404 returned error can't find the container with id 5fb2d706bc4a36a5dba45cc00ccb612938ceff742c167eaa72b1ab10256c7913
Nov 02 21:09:23.179152 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:23.179103    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c95c57675-l5zgg" event={"ID":"15c573c4-d45e-4414-96c2-0a899e3df524","Type":"ContainerStarted","Data":"9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1"}
Nov 02 21:09:23.179539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:23.179193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c95c57675-l5zgg" event={"ID":"15c573c4-d45e-4414-96c2-0a899e3df524","Type":"ContainerStarted","Data":"5fb2d706bc4a36a5dba45cc00ccb612938ceff742c167eaa72b1ab10256c7913"}
Nov 02 21:09:23.288417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:23.288368    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c95c57675-l5zgg" podStartSLOduration=2.288351897 podStartE2EDuration="2.288351897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:23.287532864 +0000 UTC m=+50.551069181" watchObservedRunningTime="2026-11-02 21:09:23.288351897 +0000 UTC m=+50.551888208"
Nov 02 21:09:32.132297 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:32.132256    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:32.132909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:32.132336    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:32.137376 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:32.137351    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:32.219767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:32.219738    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:09:32.383130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:32.383027    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b498f8d68-nhj8d"]
Nov 02 21:09:33.225352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:33.225309    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-wktvh" event={"ID":"fafaae0a-b2a1-487f-968c-4874767897eb","Type":"ContainerStarted","Data":"0da36c8885b144a92431452e44b0b350c4575c67307849908fb0844b3855f9ae"}
Nov 02 21:09:33.332251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:33.332190    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-wktvh" podStartSLOduration=2.179387276 podStartE2EDuration="27.332172248s" totalImagesPullingTime="25.152784972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:33.332040963 +0000 UTC m=+60.595577281" watchObservedRunningTime="2026-11-02 21:09:33.332172248 +0000 UTC m=+60.595708569"
Nov 02 21:09:34.229042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:34.229005    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:34.248593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:34.248557    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-wktvh"
Nov 02 21:09:44.113023 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:09:44.112979    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:09:53.181680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.181630    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77b898697b-x764j"]
Nov 02 21:09:53.186773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.186746    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.244394 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.244347    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77b898697b-x764j"]
Nov 02 21:09:53.294490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.294461    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-service-ca\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.294490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.294496    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837fb3ac-dbe9-490a-aa31-40ec1e685d39-tmp\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.294721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.294580    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-serving-cert\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.294721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.294625    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-oauth-config\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.294721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.294641    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-trusted-ca-bundle\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.294721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.294690    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-oauth-serving-cert\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.294852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.294754    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-config\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.294852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.294781    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjnh\" (UniqueName: \"kubernetes.io/projected/837fb3ac-dbe9-490a-aa31-40ec1e685d39-kube-api-access-kfjnh\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.395413 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.395372    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-serving-cert\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.395413 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.395415    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-oauth-config\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.395689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.395431    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-trusted-ca-bundle\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.395689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.395446    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-oauth-serving-cert\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.395689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.395466    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-config\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.395689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.395632    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjnh\" (UniqueName: \"kubernetes.io/projected/837fb3ac-dbe9-490a-aa31-40ec1e685d39-kube-api-access-kfjnh\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.395886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.395719    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-service-ca\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.395886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.395761    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837fb3ac-dbe9-490a-aa31-40ec1e685d39-tmp\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.396278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.396218    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837fb3ac-dbe9-490a-aa31-40ec1e685d39-tmp\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.396278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.396255    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-config\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.396443 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.396318    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-oauth-serving-cert\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.396526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.396505    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-trusted-ca-bundle\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.396573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.396544    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-service-ca\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.397993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.397969    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-oauth-config\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.398079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.398042    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-serving-cert\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.449729 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.449653    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjnh\" (UniqueName: \"kubernetes.io/projected/837fb3ac-dbe9-490a-aa31-40ec1e685d39-kube-api-access-kfjnh\") pod \"console-77b898697b-x764j\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") " pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.502838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.502796    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:09:53.877528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:53.877363    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77b898697b-x764j"]
Nov 02 21:09:53.880159 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:09:53.880098    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837fb3ac_dbe9_490a_aa31_40ec1e685d39.slice/crio-3eee97991f1da761a934df8b965c04c303d45e68486e170ee33961f79c95ba94 WatchSource:0}: Error finding container 3eee97991f1da761a934df8b965c04c303d45e68486e170ee33961f79c95ba94: Status 404 returned error can't find the container with id 3eee97991f1da761a934df8b965c04c303d45e68486e170ee33961f79c95ba94
Nov 02 21:09:54.286489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.286459    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8924h-85j4m"]
Nov 02 21:09:54.296856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.296824    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77b898697b-x764j" event={"ID":"837fb3ac-dbe9-490a-aa31-40ec1e685d39","Type":"ContainerStarted","Data":"7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0"}
Nov 02 21:09:54.297024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.296872    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77b898697b-x764j" event={"ID":"837fb3ac-dbe9-490a-aa31-40ec1e685d39","Type":"ContainerStarted","Data":"3eee97991f1da761a934df8b965c04c303d45e68486e170ee33961f79c95ba94"}
Nov 02 21:09:54.297024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.296885    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-8924h-85j4m" podUID="283a86d8-9e77-4f36-8fa9-0bd380ec043f" containerName="insights-gathering" containerID="cri-o://123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9" gracePeriod=30
Nov 02 21:09:54.511397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.511346    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77b898697b-x764j" podStartSLOduration=1.5113313860000002 podStartE2EDuration="1.511331386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:54.510493382 +0000 UTC m=+81.774029702" watchObservedRunningTime="2026-11-02 21:09:54.511331386 +0000 UTC m=+81.774867701"
Nov 02 21:09:54.549451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.549424    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:09:54.607247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.607206    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v\" (UniqueName: \"kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v\") pod \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") "
Nov 02 21:09:54.607451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.607266    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path\") pod \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") "
Nov 02 21:09:54.607451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.607321    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle\") pod \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") "
Nov 02 21:09:54.607451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.607376    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle\") pod \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\" (UID: \"283a86d8-9e77-4f36-8fa9-0bd380ec043f\") "
Nov 02 21:09:54.607714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.607691    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path" pod "283a86d8-9e77-4f36-8fa9-0bd380ec043f" (UID: "283a86d8-9e77-4f36-8fa9-0bd380ec043f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:09:54.607788 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.607746    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle" pod "283a86d8-9e77-4f36-8fa9-0bd380ec043f" (UID: "283a86d8-9e77-4f36-8fa9-0bd380ec043f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:09:54.607827 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.607792    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle" pod "283a86d8-9e77-4f36-8fa9-0bd380ec043f" (UID: "283a86d8-9e77-4f36-8fa9-0bd380ec043f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:09:54.609580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.609557    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v" pod "283a86d8-9e77-4f36-8fa9-0bd380ec043f" (UID: "283a86d8-9e77-4f36-8fa9-0bd380ec043f"). InnerVolumeSpecName "kube-api-access-njx2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:09:54.707989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.707949    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-trusted-ca-bundle\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:54.707989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.707983    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/283a86d8-9e77-4f36-8fa9-0bd380ec043f-service-ca-bundle\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:54.707989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.707994    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njx2v\" (UniqueName: \"kubernetes.io/projected/283a86d8-9e77-4f36-8fa9-0bd380ec043f-kube-api-access-njx2v\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:54.708250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:54.708005    4528 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/283a86d8-9e77-4f36-8fa9-0bd380ec043f-archives-path\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:55.301983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:55.301944    4528 generic.go:356] "Generic (PLEG): container finished" podID="283a86d8-9e77-4f36-8fa9-0bd380ec043f" containerID="123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9" exitCode=2
Nov 02 21:09:55.302607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:55.302025    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8924h-85j4m"
Nov 02 21:09:55.302607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:55.302026    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8924h-85j4m" event={"ID":"283a86d8-9e77-4f36-8fa9-0bd380ec043f","Type":"ContainerDied","Data":"123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9"}
Nov 02 21:09:55.302607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:55.302170    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8924h-85j4m" event={"ID":"283a86d8-9e77-4f36-8fa9-0bd380ec043f","Type":"ContainerDied","Data":"e3e9926d4be3dcd486d78f7b6a6d146e486b71c4bf81a33f1349044e895298f4"}
Nov 02 21:09:55.302607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:55.302197    4528 scope.go:122] "RemoveContainer" containerID="123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9"
Nov 02 21:09:55.309850 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:55.309832    4528 scope.go:122] "RemoveContainer" containerID="123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9"
Nov 02 21:09:55.310156 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:09:55.310129    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9\": container with ID starting with 123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9 not found: ID does not exist" containerID="123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9"
Nov 02 21:09:55.310268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:55.310166    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9"} err="failed to get container status \"123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9\": rpc error: code = NotFound desc = could not find container \"123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9\": container with ID starting with 123a5ebde8dff92de269dd7442b3b6c0e3ac339906e02a03a0273cc5436c83f9 not found: ID does not exist"
Nov 02 21:09:55.455941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:55.455908    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-8924h-85j4m"]
Nov 02 21:09:55.502438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:55.502400    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-8924h-85j4m"]
Nov 02 21:09:56.910996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:56.910961    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283a86d8-9e77-4f36-8fa9-0bd380ec043f" path="/var/lib/kubelet/pods/283a86d8-9e77-4f36-8fa9-0bd380ec043f/volumes"
Nov 02 21:09:57.407844 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.407774    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7b498f8d68-nhj8d" podUID="f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" containerName="console" containerID="cri-o://dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0" gracePeriod=15
Nov 02 21:09:57.695890 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.695860    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:57.731166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731133    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-config\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-config\") pod \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") "
Nov 02 21:09:57.731276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731207    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-serving-cert\") pod \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") "
Nov 02 21:09:57.731276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731257    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-service-ca\") pod \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") "
Nov 02 21:09:57.731350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731287    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-oauth-config\") pod \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") "
Nov 02 21:09:57.731350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731320    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-kube-api-access-pfcmc\" (UniqueName: \"kubernetes.io/projected/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-kube-api-access-pfcmc\") pod \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") "
Nov 02 21:09:57.731455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731370    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-tmp\") pod \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") "
Nov 02 21:09:57.731455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731425    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-oauth-serving-cert\") pod \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\" (UID: \"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e\") "
Nov 02 21:09:57.731556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731539    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-config" pod "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" (UID: "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:09:57.731750 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731713    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-tmp" pod "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" (UID: "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:09:57.731750 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731732    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-service-ca" pod "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" (UID: "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:09:57.731941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731792    4528 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-config\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:57.731941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.731829    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-oauth-serving-cert" pod "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" (UID: "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:09:57.733498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.733474    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-oauth-config" pod "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" (UID: "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:09:57.733594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.733538    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-serving-cert" pod "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" (UID: "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:09:57.733642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.733592    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-kube-api-access-pfcmc" pod "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" (UID: "f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e"). InnerVolumeSpecName "kube-api-access-pfcmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:09:57.832302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.832268    4528 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-tmp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:57.832302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.832300    4528 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-oauth-serving-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:57.832302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.832310    4528 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-serving-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:57.832538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.832320    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:57.832538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.832329    4528 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-console-oauth-config\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:57.832538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:57.832339    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfcmc\" (UniqueName: \"kubernetes.io/projected/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e-kube-api-access-pfcmc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:09:58.314718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.314683    4528 generic.go:356] "Generic (PLEG): container finished" podID="f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" containerID="dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0" exitCode=0
Nov 02 21:09:58.315238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.314736    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b498f8d68-nhj8d"
Nov 02 21:09:58.315238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.314736    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b498f8d68-nhj8d" event={"ID":"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e","Type":"ContainerDied","Data":"dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0"}
Nov 02 21:09:58.315238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.314861    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b498f8d68-nhj8d" event={"ID":"f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e","Type":"ContainerDied","Data":"741410a186a62634ec529fd075b385c5e8505ef6c1d108942bf549ada860d630"}
Nov 02 21:09:58.315238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.314882    4528 scope.go:122] "RemoveContainer" containerID="dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0"
Nov 02 21:09:58.323049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.322995    4528 scope.go:122] "RemoveContainer" containerID="dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0"
Nov 02 21:09:58.323357 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:09:58.323337    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0\": container with ID starting with dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0 not found: ID does not exist" containerID="dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0"
Nov 02 21:09:58.323437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.323366    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0"} err="failed to get container status \"dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0\": rpc error: code = NotFound desc = could not find container \"dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0\": container with ID starting with dc6949c598fcd9f6786486397455e3742f90b58081df3ab94aaa464da15859e0 not found: ID does not exist"
Nov 02 21:09:58.468176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.468135    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b498f8d68-nhj8d"]
Nov 02 21:09:58.514579 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.514547    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b498f8d68-nhj8d"]
Nov 02 21:09:58.915334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:09:58.915300    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" path="/var/lib/kubelet/pods/f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e/volumes"
Nov 02 21:10:03.503232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:03.503185    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:10:03.503714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:03.503300    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:10:03.507932 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:03.507910    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:10:04.340320 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:04.340288    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:10:04.496630 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:04.496598    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c95c57675-l5zgg"]
Nov 02 21:10:14.113088 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:10:14.113039    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:10:29.515818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.515777    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c95c57675-l5zgg" podUID="15c573c4-d45e-4414-96c2-0a899e3df524" containerName="console" containerID="cri-o://9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1" gracePeriod=15
Nov 02 21:10:29.758478 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.758453    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:10:29.891923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.891838    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-console-config\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-console-config\") pod \"15c573c4-d45e-4414-96c2-0a899e3df524\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") "
Nov 02 21:10:29.891923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.891887    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c573c4-d45e-4414-96c2-0a899e3df524-tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c573c4-d45e-4414-96c2-0a899e3df524-tmp\") pod \"15c573c4-d45e-4414-96c2-0a899e3df524\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") "
Nov 02 21:10:29.891923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.891913    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-trusted-ca-bundle\") pod \"15c573c4-d45e-4414-96c2-0a899e3df524\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") "
Nov 02 21:10:29.892199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.891936    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-oauth-config\") pod \"15c573c4-d45e-4414-96c2-0a899e3df524\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") "
Nov 02 21:10:29.892199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892007    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-service-ca\") pod \"15c573c4-d45e-4414-96c2-0a899e3df524\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") "
Nov 02 21:10:29.892199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892048    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-oauth-serving-cert\") pod \"15c573c4-d45e-4414-96c2-0a899e3df524\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") "
Nov 02 21:10:29.892199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892075    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-serving-cert\") pod \"15c573c4-d45e-4414-96c2-0a899e3df524\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") "
Nov 02 21:10:29.892199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892125    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c573c4-d45e-4414-96c2-0a899e3df524-kube-api-access-p9h5z\" (UniqueName: \"kubernetes.io/projected/15c573c4-d45e-4414-96c2-0a899e3df524-kube-api-access-p9h5z\") pod \"15c573c4-d45e-4414-96c2-0a899e3df524\" (UID: \"15c573c4-d45e-4414-96c2-0a899e3df524\") "
Nov 02 21:10:29.892460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892208    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c573c4-d45e-4414-96c2-0a899e3df524-tmp" pod "15c573c4-d45e-4414-96c2-0a899e3df524" (UID: "15c573c4-d45e-4414-96c2-0a899e3df524"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:10:29.892460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892311    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-console-config" pod "15c573c4-d45e-4414-96c2-0a899e3df524" (UID: "15c573c4-d45e-4414-96c2-0a899e3df524"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:10:29.892460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892332    4528 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c573c4-d45e-4414-96c2-0a899e3df524-tmp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:29.892460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892447    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-trusted-ca-bundle" pod "15c573c4-d45e-4414-96c2-0a899e3df524" (UID: "15c573c4-d45e-4414-96c2-0a899e3df524"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:10:29.892659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892520    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-oauth-serving-cert" pod "15c573c4-d45e-4414-96c2-0a899e3df524" (UID: "15c573c4-d45e-4414-96c2-0a899e3df524"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:10:29.892659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.892557    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-service-ca" pod "15c573c4-d45e-4414-96c2-0a899e3df524" (UID: "15c573c4-d45e-4414-96c2-0a899e3df524"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:10:29.894235 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.894210    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-oauth-config" pod "15c573c4-d45e-4414-96c2-0a899e3df524" (UID: "15c573c4-d45e-4414-96c2-0a899e3df524"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:10:29.894350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.894237    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-serving-cert" pod "15c573c4-d45e-4414-96c2-0a899e3df524" (UID: "15c573c4-d45e-4414-96c2-0a899e3df524"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:10:29.894350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.894311    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c573c4-d45e-4414-96c2-0a899e3df524-kube-api-access-p9h5z" pod "15c573c4-d45e-4414-96c2-0a899e3df524" (UID: "15c573c4-d45e-4414-96c2-0a899e3df524"). InnerVolumeSpecName "kube-api-access-p9h5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:10:29.993065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.993024    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9h5z\" (UniqueName: \"kubernetes.io/projected/15c573c4-d45e-4414-96c2-0a899e3df524-kube-api-access-p9h5z\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:29.993065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.993058    4528 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-console-config\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:29.993065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.993068    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-trusted-ca-bundle\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:29.993065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.993077    4528 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-oauth-config\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:29.993347 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.993086    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:29.993347 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.993095    4528 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/15c573c4-d45e-4414-96c2-0a899e3df524-oauth-serving-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:29.993347 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:29.993104    4528 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/15c573c4-d45e-4414-96c2-0a899e3df524-console-serving-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:10:30.422619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.422588    4528 generic.go:356] "Generic (PLEG): container finished" podID="15c573c4-d45e-4414-96c2-0a899e3df524" containerID="9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1" exitCode=0
Nov 02 21:10:30.422789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.422628    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c95c57675-l5zgg" event={"ID":"15c573c4-d45e-4414-96c2-0a899e3df524","Type":"ContainerDied","Data":"9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1"}
Nov 02 21:10:30.422789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.422638    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c95c57675-l5zgg"
Nov 02 21:10:30.422789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.422666    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c95c57675-l5zgg" event={"ID":"15c573c4-d45e-4414-96c2-0a899e3df524","Type":"ContainerDied","Data":"5fb2d706bc4a36a5dba45cc00ccb612938ceff742c167eaa72b1ab10256c7913"}
Nov 02 21:10:30.422789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.422685    4528 scope.go:122] "RemoveContainer" containerID="9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1"
Nov 02 21:10:30.430494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.430474    4528 scope.go:122] "RemoveContainer" containerID="9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1"
Nov 02 21:10:30.430790 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:10:30.430768    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1\": container with ID starting with 9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1 not found: ID does not exist" containerID="9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1"
Nov 02 21:10:30.430845 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.430799    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1"} err="failed to get container status \"9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1\": rpc error: code = NotFound desc = could not find container \"9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1\": container with ID starting with 9980e614803ebf2edc94dba432d11d5bf206c74a90040e7878e642c8f6d233d1 not found: ID does not exist"
Nov 02 21:10:30.575410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.575379    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c95c57675-l5zgg"]
Nov 02 21:10:30.622044 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.622012    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c95c57675-l5zgg"]
Nov 02 21:10:30.909941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:10:30.909906    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c573c4-d45e-4414-96c2-0a899e3df524" path="/var/lib/kubelet/pods/15c573c4-d45e-4414-96c2-0a899e3df524/volumes"
Nov 02 21:10:44.112223 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:10:44.112185    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:11:02.018741 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.018706    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c677c9dc8-t4bsc"]
Nov 02 21:11:02.020930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.019352    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" containerName="console"
Nov 02 21:11:02.020930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.019381    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" containerName="console"
Nov 02 21:11:02.020930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.019428    4528 cpu_manager.go:414] "containerMap: removing container" podUID="15c573c4-d45e-4414-96c2-0a899e3df524" containerName="console"
Nov 02 21:11:02.020930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.019436    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c573c4-d45e-4414-96c2-0a899e3df524" containerName="console"
Nov 02 21:11:02.020930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.019488    4528 cpu_manager.go:414] "containerMap: removing container" podUID="283a86d8-9e77-4f36-8fa9-0bd380ec043f" containerName="insights-gathering"
Nov 02 21:11:02.020930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.019496    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283a86d8-9e77-4f36-8fa9-0bd380ec043f" containerName="insights-gathering"
Nov 02 21:11:02.020930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.019651    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c573c4-d45e-4414-96c2-0a899e3df524" containerName="console"
Nov 02 21:11:02.020930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.019664    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="283a86d8-9e77-4f36-8fa9-0bd380ec043f" containerName="insights-gathering"
Nov 02 21:11:02.020930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.019682    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f90cdc-dc3c-4ab7-b5d4-34f03992a48e" containerName="console"
Nov 02 21:11:02.023201 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.023171    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.119209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.119174    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c677c9dc8-t4bsc"]
Nov 02 21:11:02.126268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.126243    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsskz\" (UniqueName: \"kubernetes.io/projected/83fa44da-6c11-4058-9f5e-f92087775796-kube-api-access-vsskz\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.126393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.126284    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-service-ca\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.126393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.126348    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83fa44da-6c11-4058-9f5e-f92087775796-tmp\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.126393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.126387    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-trusted-ca-bundle\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.126512 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.126419    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83fa44da-6c11-4058-9f5e-f92087775796-console-serving-cert\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.126512 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.126475    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83fa44da-6c11-4058-9f5e-f92087775796-console-oauth-config\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.126512 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.126501    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-console-config\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.126651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.126529    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-oauth-serving-cert\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.227254 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.227221    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83fa44da-6c11-4058-9f5e-f92087775796-tmp\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.227254 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.227254    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-trusted-ca-bundle\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.227487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.227290    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83fa44da-6c11-4058-9f5e-f92087775796-console-serving-cert\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.227487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.227314    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83fa44da-6c11-4058-9f5e-f92087775796-console-oauth-config\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.227487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.227335    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-console-config\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.227487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.227353    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-oauth-serving-cert\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.227487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.227406    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsskz\" (UniqueName: \"kubernetes.io/projected/83fa44da-6c11-4058-9f5e-f92087775796-kube-api-access-vsskz\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.227487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.227444    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-service-ca\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.227789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.227682    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83fa44da-6c11-4058-9f5e-f92087775796-tmp\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.228162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.228139    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-service-ca\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.228162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.228155    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-console-config\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.228362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.228339    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-oauth-serving-cert\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.228402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.228382    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83fa44da-6c11-4058-9f5e-f92087775796-trusted-ca-bundle\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.229925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.229901    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83fa44da-6c11-4058-9f5e-f92087775796-console-oauth-config\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.230014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.229946    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83fa44da-6c11-4058-9f5e-f92087775796-console-serving-cert\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.280747 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.280688    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsskz\" (UniqueName: \"kubernetes.io/projected/83fa44da-6c11-4058-9f5e-f92087775796-kube-api-access-vsskz\") pod \"console-7c677c9dc8-t4bsc\" (UID: \"83fa44da-6c11-4058-9f5e-f92087775796\") " pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.333952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.333928    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:02.697216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:02.697146    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c677c9dc8-t4bsc"]
Nov 02 21:11:02.700150 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:11:02.700102    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fa44da_6c11_4058_9f5e_f92087775796.slice/crio-5fa6c3a0718c88fc960bde184cd184ecb118fb41321630f850c4db90eb5c8802 WatchSource:0}: Error finding container 5fa6c3a0718c88fc960bde184cd184ecb118fb41321630f850c4db90eb5c8802: Status 404 returned error can't find the container with id 5fa6c3a0718c88fc960bde184cd184ecb118fb41321630f850c4db90eb5c8802
Nov 02 21:11:03.526835 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:03.526804    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c677c9dc8-t4bsc" event={"ID":"83fa44da-6c11-4058-9f5e-f92087775796","Type":"ContainerStarted","Data":"9a8164bbb42574da4a206fb46a3c5d2e60c2746d954ec9fad18884cf150637ae"}
Nov 02 21:11:03.527236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:03.526851    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c677c9dc8-t4bsc" event={"ID":"83fa44da-6c11-4058-9f5e-f92087775796","Type":"ContainerStarted","Data":"5fa6c3a0718c88fc960bde184cd184ecb118fb41321630f850c4db90eb5c8802"}
Nov 02 21:11:03.682183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:03.682135    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c677c9dc8-t4bsc" podStartSLOduration=2.68210136 podStartE2EDuration="2.68210136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:11:03.635384847 +0000 UTC m=+150.898921160" watchObservedRunningTime="2026-11-02 21:11:03.68210136 +0000 UTC m=+150.945637677"
Nov 02 21:11:12.334778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:12.334741    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:12.335206 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:12.334810    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:12.339724 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:12.339701    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:12.562236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:12.562209    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c677c9dc8-t4bsc"
Nov 02 21:11:12.696953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:12.696877    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77b898697b-x764j"]
Nov 02 21:11:14.111675 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:11:14.111644    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:11:37.716070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:37.716008    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-77b898697b-x764j" podUID="837fb3ac-dbe9-490a-aa31-40ec1e685d39" containerName="console" containerID="cri-o://7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0" gracePeriod=15
Nov 02 21:11:37.956791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:37.956760    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:11:38.102895 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.102853    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-config\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-config\") pod \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") "
Nov 02 21:11:38.102895 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.102901    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-service-ca\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-service-ca\") pod \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") "
Nov 02 21:11:38.103180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.102930    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837fb3ac-dbe9-490a-aa31-40ec1e685d39-kube-api-access-kfjnh\" (UniqueName: \"kubernetes.io/projected/837fb3ac-dbe9-490a-aa31-40ec1e685d39-kube-api-access-kfjnh\") pod \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") "
Nov 02 21:11:38.103180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.102967    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837fb3ac-dbe9-490a-aa31-40ec1e685d39-tmp\" (UniqueName: \"kubernetes.io/empty-dir/837fb3ac-dbe9-490a-aa31-40ec1e685d39-tmp\") pod \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") "
Nov 02 21:11:38.103180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103151    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-oauth-config\") pod \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") "
Nov 02 21:11:38.103338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103210    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-trusted-ca-bundle\") pod \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") "
Nov 02 21:11:38.103338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103258    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-serving-cert\") pod \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") "
Nov 02 21:11:38.103338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103298    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-oauth-serving-cert\") pod \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\" (UID: \"837fb3ac-dbe9-490a-aa31-40ec1e685d39\") "
Nov 02 21:11:38.103338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103297    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-service-ca" pod "837fb3ac-dbe9-490a-aa31-40ec1e685d39" (UID: "837fb3ac-dbe9-490a-aa31-40ec1e685d39"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:11:38.103537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103344    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837fb3ac-dbe9-490a-aa31-40ec1e685d39-tmp" pod "837fb3ac-dbe9-490a-aa31-40ec1e685d39" (UID: "837fb3ac-dbe9-490a-aa31-40ec1e685d39"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:11:38.103537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103353    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-config" pod "837fb3ac-dbe9-490a-aa31-40ec1e685d39" (UID: "837fb3ac-dbe9-490a-aa31-40ec1e685d39"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:11:38.103537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103508    4528 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/837fb3ac-dbe9-490a-aa31-40ec1e685d39-tmp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:38.103537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103533    4528 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-config\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:38.103743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103547    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:38.103743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103621    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-trusted-ca-bundle" pod "837fb3ac-dbe9-490a-aa31-40ec1e685d39" (UID: "837fb3ac-dbe9-490a-aa31-40ec1e685d39"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:11:38.103743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.103728    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-oauth-serving-cert" pod "837fb3ac-dbe9-490a-aa31-40ec1e685d39" (UID: "837fb3ac-dbe9-490a-aa31-40ec1e685d39"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:11:38.105174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.105150    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-oauth-config" pod "837fb3ac-dbe9-490a-aa31-40ec1e685d39" (UID: "837fb3ac-dbe9-490a-aa31-40ec1e685d39"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:11:38.105282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.105167    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837fb3ac-dbe9-490a-aa31-40ec1e685d39-kube-api-access-kfjnh" pod "837fb3ac-dbe9-490a-aa31-40ec1e685d39" (UID: "837fb3ac-dbe9-490a-aa31-40ec1e685d39"). InnerVolumeSpecName "kube-api-access-kfjnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:11:38.105340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.105308    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-serving-cert" pod "837fb3ac-dbe9-490a-aa31-40ec1e685d39" (UID: "837fb3ac-dbe9-490a-aa31-40ec1e685d39"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:11:38.204468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.204417    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfjnh\" (UniqueName: \"kubernetes.io/projected/837fb3ac-dbe9-490a-aa31-40ec1e685d39-kube-api-access-kfjnh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:38.204468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.204461    4528 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-oauth-config\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:38.204468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.204471    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-trusted-ca-bundle\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:38.204468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.204481    4528 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837fb3ac-dbe9-490a-aa31-40ec1e685d39-console-serving-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:38.204468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.204489    4528 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837fb3ac-dbe9-490a-aa31-40ec1e685d39-oauth-serving-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:11:38.637459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.637423    4528 generic.go:356] "Generic (PLEG): container finished" podID="837fb3ac-dbe9-490a-aa31-40ec1e685d39" containerID="7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0" exitCode=0
Nov 02 21:11:38.637657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.637477    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77b898697b-x764j"
Nov 02 21:11:38.637657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.637506    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77b898697b-x764j" event={"ID":"837fb3ac-dbe9-490a-aa31-40ec1e685d39","Type":"ContainerDied","Data":"7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0"}
Nov 02 21:11:38.637657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.637559    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77b898697b-x764j" event={"ID":"837fb3ac-dbe9-490a-aa31-40ec1e685d39","Type":"ContainerDied","Data":"3eee97991f1da761a934df8b965c04c303d45e68486e170ee33961f79c95ba94"}
Nov 02 21:11:38.637657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.637579    4528 scope.go:122] "RemoveContainer" containerID="7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0"
Nov 02 21:11:38.645960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.645943    4528 scope.go:122] "RemoveContainer" containerID="7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0"
Nov 02 21:11:38.646349 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:11:38.646331    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0\": container with ID starting with 7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0 not found: ID does not exist" containerID="7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0"
Nov 02 21:11:38.646420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.646362    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0"} err="failed to get container status \"7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0\": rpc error: code = NotFound desc = could not find container \"7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0\": container with ID starting with 7887d29854bb3649ad4a648c80af0176b6d3d2ac972c908b106e2ced06a87dd0 not found: ID does not exist"
Nov 02 21:11:38.790166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.790137    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77b898697b-x764j"]
Nov 02 21:11:38.836379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.836349    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77b898697b-x764j"]
Nov 02 21:11:38.911946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:11:38.911851    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837fb3ac-dbe9-490a-aa31-40ec1e685d39" path="/var/lib/kubelet/pods/837fb3ac-dbe9-490a-aa31-40ec1e685d39/volumes"
Nov 02 21:11:44.111940 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:11:44.111897    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:12:14.111613 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:12:14.111573    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:12:44.112182 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:12:44.112137    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:12:49.924024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.923997    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"]
Nov 02 21:12:49.924607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.924284    4528 cpu_manager.go:414] "containerMap: removing container" podUID="837fb3ac-dbe9-490a-aa31-40ec1e685d39" containerName="console"
Nov 02 21:12:49.924607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.924296    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837fb3ac-dbe9-490a-aa31-40ec1e685d39" containerName="console"
Nov 02 21:12:49.924607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.924349    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="837fb3ac-dbe9-490a-aa31-40ec1e685d39" containerName="console"
Nov 02 21:12:49.927199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.927182    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:49.978460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.978426    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 21:12:49.978460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.978441    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 21:12:49.978674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.978500    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 21:12:49.978674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.978511    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 21:12:49.978674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:49.978574    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 21:12:50.024673 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.024641    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n4jrm\""
Nov 02 21:12:50.025698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.025673    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"]
Nov 02 21:12:50.037049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.037020    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.037203 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.037052    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz28m\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-kube-api-access-dz28m\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.037203 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.037081    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.037203 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.037152    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.037315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.037279    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.037315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.037309    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.037394 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.037344    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5c8a40-b73c-4379-aba6-869ed93e4384-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.079731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.079703    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 21:12:50.138028 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.137993    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.138028 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.138031    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz28m\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-kube-api-access-dz28m\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.138311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.138067    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.138311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.138096    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.138311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.138156    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.138311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.138181    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.138311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.138221    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5c8a40-b73c-4379-aba6-869ed93e4384-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.138861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.138832    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.140589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.140563    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.140698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.140583    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.140742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.140732    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5c8a40-b73c-4379-aba6-869ed93e4384-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.140949 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.140926    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.192159 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.192078    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.192542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.192520    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz28m\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-kube-api-access-dz28m\") pod \"istiod-openshift-gateway-7876bfcb74-z4z9v\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.242503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.242466    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:50.651241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.651200    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"]
Nov 02 21:12:50.653908 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:12:50.653878    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5c8a40_b73c_4379_aba6_869ed93e4384.slice/crio-cc3ad0dd339f7541dd7782f38cb9bacb1260afcf8acb9ed54f6fb4a5ba878a2c WatchSource:0}: Error finding container cc3ad0dd339f7541dd7782f38cb9bacb1260afcf8acb9ed54f6fb4a5ba878a2c: Status 404 returned error can't find the container with id cc3ad0dd339f7541dd7782f38cb9bacb1260afcf8acb9ed54f6fb4a5ba878a2c
Nov 02 21:12:50.864392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:50.864354    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v" event={"ID":"5f5c8a40-b73c-4379-aba6-869ed93e4384","Type":"ContainerStarted","Data":"cc3ad0dd339f7541dd7782f38cb9bacb1260afcf8acb9ed54f6fb4a5ba878a2c"}
Nov 02 21:12:55.009450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:55.008741    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:12:55.009450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:55.008798    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:12:55.883174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:55.883126    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v" event={"ID":"5f5c8a40-b73c-4379-aba6-869ed93e4384","Type":"ContainerStarted","Data":"cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d"}
Nov 02 21:12:55.883346 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:55.883233    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:12:56.085382 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:56.085329    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v" podStartSLOduration=2.733294329 podStartE2EDuration="7.085316034s" totalImagesPullingTime="4.352021705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:56.038781997 +0000 UTC m=+263.302318312" watchObservedRunningTime="2026-11-02 21:12:56.085316034 +0000 UTC m=+263.348852352"
Nov 02 21:12:56.888007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:56.887976    4528 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-z4z9v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:12:56.888183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:56.888039    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v" podUID="5f5c8a40-b73c-4379-aba6-869ed93e4384" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:12:59.888059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:12:59.888028    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:13:14.112404 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:13:14.112364    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:13:44.111887 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:13:44.111849    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:13:51.670695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.670660    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"]
Nov 02 21:13:51.673723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.673704    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.686983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.686959    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c1d75c6-3d49-43e5-961a-099b20f0aad2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.687091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.687011    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c1d75c6-3d49-43e5-961a-099b20f0aad2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.687156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.687129    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c1d75c6-3d49-43e5-961a-099b20f0aad2-manifests\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.687198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.687178    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c1d75c6-3d49-43e5-961a-099b20f0aad2-tmp\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.687238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.687208    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrv4\" (UniqueName: \"kubernetes.io/projected/1c1d75c6-3d49-43e5-961a-099b20f0aad2-kube-api-access-jzrv4\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.687270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.687249    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c1d75c6-3d49-43e5-961a-099b20f0aad2-metrics-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.787594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.787565    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c1d75c6-3d49-43e5-961a-099b20f0aad2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.787771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.787604    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c1d75c6-3d49-43e5-961a-099b20f0aad2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.787771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.787633    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c1d75c6-3d49-43e5-961a-099b20f0aad2-manifests\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.787771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.787656    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c1d75c6-3d49-43e5-961a-099b20f0aad2-tmp\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.787771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.787680    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrv4\" (UniqueName: \"kubernetes.io/projected/1c1d75c6-3d49-43e5-961a-099b20f0aad2-kube-api-access-jzrv4\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.787771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.787706    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c1d75c6-3d49-43e5-961a-099b20f0aad2-metrics-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.787961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.787942    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c1d75c6-3d49-43e5-961a-099b20f0aad2-manifests\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.789849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.789819    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c1d75c6-3d49-43e5-961a-099b20f0aad2-tmp\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.812649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.812623    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 21:13:51.812805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.812723    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 21:13:51.812805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.812751    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 21:13:51.812925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.812811    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 21:13:51.812976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.812962    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rhszk\""
Nov 02 21:13:51.820292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.820274    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c1d75c6-3d49-43e5-961a-099b20f0aad2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.820453 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.820432    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c1d75c6-3d49-43e5-961a-099b20f0aad2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.823629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.823612    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 21:13:51.830656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.830633    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c1d75c6-3d49-43e5-961a-099b20f0aad2-metrics-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.861999 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.861972    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrv4\" (UniqueName: \"kubernetes.io/projected/1c1d75c6-3d49-43e5-961a-099b20f0aad2-kube-api-access-jzrv4\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-w4zhb\" (UID: \"1c1d75c6-3d49-43e5-961a-099b20f0aad2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:51.936381 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.936298    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"]
Nov 02 21:13:51.988073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:51.988044    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:52.460893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:52.460730    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"]
Nov 02 21:13:52.463597 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:13:52.463565    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1d75c6_3d49_43e5_961a_099b20f0aad2.slice/crio-527facf0c569c8495e2249d1bcfe777b873ab73be4ecdeb855b9253e8f1b56ed WatchSource:0}: Error finding container 527facf0c569c8495e2249d1bcfe777b873ab73be4ecdeb855b9253e8f1b56ed: Status 404 returned error can't find the container with id 527facf0c569c8495e2249d1bcfe777b873ab73be4ecdeb855b9253e8f1b56ed
Nov 02 21:13:52.465300 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:52.465282    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:13:53.065287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:53.065250    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb" event={"ID":"1c1d75c6-3d49-43e5-961a-099b20f0aad2","Type":"ContainerStarted","Data":"527facf0c569c8495e2249d1bcfe777b873ab73be4ecdeb855b9253e8f1b56ed"}
Nov 02 21:13:57.083311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:57.083276    4528 generic.go:356] "Generic (PLEG): container finished" podID="1c1d75c6-3d49-43e5-961a-099b20f0aad2" containerID="33a88136d369df5ff9a72d79b66fa415996d5e8239572d0ae19e1612abb3d440" exitCode=0
Nov 02 21:13:57.083667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:57.083318    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb" event={"ID":"1c1d75c6-3d49-43e5-961a-099b20f0aad2","Type":"ContainerDied","Data":"33a88136d369df5ff9a72d79b66fa415996d5e8239572d0ae19e1612abb3d440"}
Nov 02 21:13:58.088460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:58.088425    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb" event={"ID":"1c1d75c6-3d49-43e5-961a-099b20f0aad2","Type":"ContainerStarted","Data":"5e20be6d20d453f9823f3455d1a4b8592a5bab9944965ae075535eb09af94cde"}
Nov 02 21:13:58.088929 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:58.088546    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:13:58.256336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:13:58.256241    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb" podStartSLOduration=1.99610757 podStartE2EDuration="7.256228312s" totalImagesPullingTime="4.782570401s" totalInitContainerRuntime="477.550341ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:58.25482163 +0000 UTC m=+325.518357999" watchObservedRunningTime="2026-11-02 21:13:58.256228312 +0000 UTC m=+325.519764677"
Nov 02 21:14:09.094073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:09.094024    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-w4zhb"
Nov 02 21:14:14.111065 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:14:14.111025    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:14:23.197166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.197125    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"]
Nov 02 21:14:23.200537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.200513    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.249061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.249034    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 21:14:23.295588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.295556    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 21:14:23.295796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.295672    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-62vv5\""
Nov 02 21:14:23.296775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.296753    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"]
Nov 02 21:14:23.340528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.340497    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2zv\" (UniqueName: \"kubernetes.io/projected/cb4c26f6-5512-4b9d-9c66-0a2a45798eb3-kube-api-access-mp2zv\") pod \"kube-auth-proxy-78fbf7b575-zlh7p\" (UID: \"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.340528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.340539    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4c26f6-5512-4b9d-9c66-0a2a45798eb3-tmp\") pod \"kube-auth-proxy-78fbf7b575-zlh7p\" (UID: \"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.340777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.340743    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4c26f6-5512-4b9d-9c66-0a2a45798eb3-tls-certs\") pod \"kube-auth-proxy-78fbf7b575-zlh7p\" (UID: \"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.441640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.441598    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4c26f6-5512-4b9d-9c66-0a2a45798eb3-tmp\") pod \"kube-auth-proxy-78fbf7b575-zlh7p\" (UID: \"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.441830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.441696    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4c26f6-5512-4b9d-9c66-0a2a45798eb3-tls-certs\") pod \"kube-auth-proxy-78fbf7b575-zlh7p\" (UID: \"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.441830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.441727    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2zv\" (UniqueName: \"kubernetes.io/projected/cb4c26f6-5512-4b9d-9c66-0a2a45798eb3-kube-api-access-mp2zv\") pod \"kube-auth-proxy-78fbf7b575-zlh7p\" (UID: \"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.443885 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.443858    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4c26f6-5512-4b9d-9c66-0a2a45798eb3-tmp\") pod \"kube-auth-proxy-78fbf7b575-zlh7p\" (UID: \"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.444205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.444184    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4c26f6-5512-4b9d-9c66-0a2a45798eb3-tls-certs\") pod \"kube-auth-proxy-78fbf7b575-zlh7p\" (UID: \"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.494881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.494817    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2zv\" (UniqueName: \"kubernetes.io/projected/cb4c26f6-5512-4b9d-9c66-0a2a45798eb3-kube-api-access-mp2zv\") pod \"kube-auth-proxy-78fbf7b575-zlh7p\" (UID: \"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.511542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.511511    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"
Nov 02 21:14:23.879948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:23.879913    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p"]
Nov 02 21:14:23.883276 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:14:23.883244    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4c26f6_5512_4b9d_9c66_0a2a45798eb3.slice/crio-05082036b2163d133e8d03506abcbf18fd534f8a03c8c785d0722f6f70b99592 WatchSource:0}: Error finding container 05082036b2163d133e8d03506abcbf18fd534f8a03c8c785d0722f6f70b99592: Status 404 returned error can't find the container with id 05082036b2163d133e8d03506abcbf18fd534f8a03c8c785d0722f6f70b99592
Nov 02 21:14:24.173659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:24.173579    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p" event={"ID":"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3","Type":"ContainerStarted","Data":"05082036b2163d133e8d03506abcbf18fd534f8a03c8c785d0722f6f70b99592"}
Nov 02 21:14:27.188246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:27.188163    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p" event={"ID":"cb4c26f6-5512-4b9d-9c66-0a2a45798eb3","Type":"ContainerStarted","Data":"631e68d1fdce40913d2c46b10d66043bc1143ee7900575402d751d39f6f47d5f"}
Nov 02 21:14:27.294778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:27.294718    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-zlh7p" podStartSLOduration=1.352707138 podStartE2EDuration="4.294699717s" totalImagesPullingTime="2.941992579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:27.293267727 +0000 UTC m=+354.556804061" watchObservedRunningTime="2026-11-02 21:14:27.294699717 +0000 UTC m=+354.558236034"
Nov 02 21:14:28.289898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.289861    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z8kws"]
Nov 02 21:14:28.296973 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.296951    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z8kws"
Nov 02 21:14:28.343402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.343360    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z8kws"]
Nov 02 21:14:28.387033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.387005    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdh2h\" (UniqueName: \"kubernetes.io/projected/89b66e11-7f22-49ad-ae94-235974f17308-kube-api-access-jdh2h\") pod \"authorino-57f7f8659f-z8kws\" (UID: \"89b66e11-7f22-49ad-ae94-235974f17308\") " pod="kuadrant-system/authorino-57f7f8659f-z8kws"
Nov 02 21:14:28.388789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.388757    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 21:14:28.388938 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.388804    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tr6zv\""
Nov 02 21:14:28.391852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.391833    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 21:14:28.488361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.488326    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdh2h\" (UniqueName: \"kubernetes.io/projected/89b66e11-7f22-49ad-ae94-235974f17308-kube-api-access-jdh2h\") pod \"authorino-57f7f8659f-z8kws\" (UID: \"89b66e11-7f22-49ad-ae94-235974f17308\") " pod="kuadrant-system/authorino-57f7f8659f-z8kws"
Nov 02 21:14:28.542902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.542831    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdh2h\" (UniqueName: \"kubernetes.io/projected/89b66e11-7f22-49ad-ae94-235974f17308-kube-api-access-jdh2h\") pod \"authorino-57f7f8659f-z8kws\" (UID: \"89b66e11-7f22-49ad-ae94-235974f17308\") " pod="kuadrant-system/authorino-57f7f8659f-z8kws"
Nov 02 21:14:28.619489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:28.619455    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z8kws"
Nov 02 21:14:29.021839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:29.021813    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z8kws"]
Nov 02 21:14:29.024305 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:14:29.024278    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89b66e11_7f22_49ad_ae94_235974f17308.slice/crio-acc4b1b4c5373cb07a4a133cf88f160e4664b61006102f57c8eb7204535ac903 WatchSource:0}: Error finding container acc4b1b4c5373cb07a4a133cf88f160e4664b61006102f57c8eb7204535ac903: Status 404 returned error can't find the container with id acc4b1b4c5373cb07a4a133cf88f160e4664b61006102f57c8eb7204535ac903
Nov 02 21:14:29.201613 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:29.201574    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z8kws" event={"ID":"89b66e11-7f22-49ad-ae94-235974f17308","Type":"ContainerStarted","Data":"acc4b1b4c5373cb07a4a133cf88f160e4664b61006102f57c8eb7204535ac903"}
Nov 02 21:14:30.436443 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.436402    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:14:30.441290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.441265    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:14:30.491231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.491196    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 21:14:30.504554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.504520    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/598f64cf-7fb5-4b62-a4ee-29802c83c4f4-config-file\") pod \"limitador-limitador-7c9b87d574-5tp5v\" (UID: \"598f64cf-7fb5-4b62-a4ee-29802c83c4f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:14:30.504711 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.504568    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72m5k\" (UniqueName: \"kubernetes.io/projected/598f64cf-7fb5-4b62-a4ee-29802c83c4f4-kube-api-access-72m5k\") pod \"limitador-limitador-7c9b87d574-5tp5v\" (UID: \"598f64cf-7fb5-4b62-a4ee-29802c83c4f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:14:30.537442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.537411    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pb7w5\""
Nov 02 21:14:30.538547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.538524    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:14:30.605890 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.605858    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/598f64cf-7fb5-4b62-a4ee-29802c83c4f4-config-file\") pod \"limitador-limitador-7c9b87d574-5tp5v\" (UID: \"598f64cf-7fb5-4b62-a4ee-29802c83c4f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:14:30.606082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.605907    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72m5k\" (UniqueName: \"kubernetes.io/projected/598f64cf-7fb5-4b62-a4ee-29802c83c4f4-kube-api-access-72m5k\") pod \"limitador-limitador-7c9b87d574-5tp5v\" (UID: \"598f64cf-7fb5-4b62-a4ee-29802c83c4f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:14:30.606649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.606630    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/598f64cf-7fb5-4b62-a4ee-29802c83c4f4-config-file\") pod \"limitador-limitador-7c9b87d574-5tp5v\" (UID: \"598f64cf-7fb5-4b62-a4ee-29802c83c4f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:14:30.659292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.659263    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72m5k\" (UniqueName: \"kubernetes.io/projected/598f64cf-7fb5-4b62-a4ee-29802c83c4f4-kube-api-access-72m5k\") pod \"limitador-limitador-7c9b87d574-5tp5v\" (UID: \"598f64cf-7fb5-4b62-a4ee-29802c83c4f4\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:14:30.754480 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.754398    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:14:30.764604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:30.764579    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:14:31.117434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:31.117341    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:14:31.119546 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:14:31.119515    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598f64cf_7fb5_4b62_a4ee_29802c83c4f4.slice/crio-13143d63720b7c600955d40a385ae29aa8b7e839a48412f7e4b19370ba4c4d5c WatchSource:0}: Error finding container 13143d63720b7c600955d40a385ae29aa8b7e839a48412f7e4b19370ba4c4d5c: Status 404 returned error can't find the container with id 13143d63720b7c600955d40a385ae29aa8b7e839a48412f7e4b19370ba4c4d5c
Nov 02 21:14:31.211031 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:31.210997    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v" event={"ID":"598f64cf-7fb5-4b62-a4ee-29802c83c4f4","Type":"ContainerStarted","Data":"13143d63720b7c600955d40a385ae29aa8b7e839a48412f7e4b19370ba4c4d5c"}
Nov 02 21:14:34.224810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:34.224544    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v" event={"ID":"598f64cf-7fb5-4b62-a4ee-29802c83c4f4","Type":"ContainerStarted","Data":"af53f172ed7b71b61a7885d36d7be717e4129abcb053c8558c724da8cc58c9f8"}
Nov 02 21:14:34.230435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:34.225552    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:14:34.230435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:34.227250    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z8kws" event={"ID":"89b66e11-7f22-49ad-ae94-235974f17308","Type":"ContainerStarted","Data":"30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f"}
Nov 02 21:14:34.333040 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:34.332989    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v" podStartSLOduration=1.75596625 podStartE2EDuration="4.332976101s" totalImagesPullingTime="2.577009851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:34.331455704 +0000 UTC m=+361.594992022" watchObservedRunningTime="2026-11-02 21:14:34.332976101 +0000 UTC m=+361.596512419"
Nov 02 21:14:34.437468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:34.437411    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-z8kws" podStartSLOduration=2.174483252 podStartE2EDuration="6.437397128s" totalImagesPullingTime="4.262913876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:34.436877217 +0000 UTC m=+361.700413534" watchObservedRunningTime="2026-11-02 21:14:34.437397128 +0000 UTC m=+361.700933445"
Nov 02 21:14:34.757898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:34.757864    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z8kws"]
Nov 02 21:14:36.234389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:36.234329    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-z8kws" podUID="89b66e11-7f22-49ad-ae94-235974f17308" containerName="authorino" containerID="cri-o://30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f" gracePeriod=30
Nov 02 21:14:36.479762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:36.479739    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z8kws"
Nov 02 21:14:36.556343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:36.556308    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b66e11-7f22-49ad-ae94-235974f17308-kube-api-access-jdh2h\" (UniqueName: \"kubernetes.io/projected/89b66e11-7f22-49ad-ae94-235974f17308-kube-api-access-jdh2h\") pod \"89b66e11-7f22-49ad-ae94-235974f17308\" (UID: \"89b66e11-7f22-49ad-ae94-235974f17308\") "
Nov 02 21:14:36.558501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:36.558477    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b66e11-7f22-49ad-ae94-235974f17308-kube-api-access-jdh2h" pod "89b66e11-7f22-49ad-ae94-235974f17308" (UID: "89b66e11-7f22-49ad-ae94-235974f17308"). InnerVolumeSpecName "kube-api-access-jdh2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:14:36.657455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:36.657417    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdh2h\" (UniqueName: \"kubernetes.io/projected/89b66e11-7f22-49ad-ae94-235974f17308-kube-api-access-jdh2h\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:14:37.238789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:37.238698    4528 generic.go:356] "Generic (PLEG): container finished" podID="89b66e11-7f22-49ad-ae94-235974f17308" containerID="30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f" exitCode=0
Nov 02 21:14:37.238789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:37.238746    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z8kws"
Nov 02 21:14:37.239331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:37.238783    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z8kws" event={"ID":"89b66e11-7f22-49ad-ae94-235974f17308","Type":"ContainerDied","Data":"30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f"}
Nov 02 21:14:37.239331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:37.238858    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z8kws" event={"ID":"89b66e11-7f22-49ad-ae94-235974f17308","Type":"ContainerDied","Data":"acc4b1b4c5373cb07a4a133cf88f160e4664b61006102f57c8eb7204535ac903"}
Nov 02 21:14:37.239331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:37.238888    4528 scope.go:122] "RemoveContainer" containerID="30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f"
Nov 02 21:14:37.246875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:37.246853    4528 scope.go:122] "RemoveContainer" containerID="30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f"
Nov 02 21:14:37.247157 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:14:37.247138    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f\": container with ID starting with 30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f not found: ID does not exist" containerID="30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f"
Nov 02 21:14:37.247209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:37.247169    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f"} err="failed to get container status \"30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f\": rpc error: code = NotFound desc = could not find container \"30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f\": container with ID starting with 30d8dfb69e6e70743fa0b9fcd56d21750f51a149027ec6fa58578e03eb87bd2f not found: ID does not exist"
Nov 02 21:14:37.394142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:37.394097    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z8kws"]
Nov 02 21:14:37.397565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:37.397541    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z8kws"]
Nov 02 21:14:38.910412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:38.910381    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b66e11-7f22-49ad-ae94-235974f17308" path="/var/lib/kubelet/pods/89b66e11-7f22-49ad-ae94-235974f17308/volumes"
Nov 02 21:14:44.111955 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:14:44.111916    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:14:45.231909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:14:45.231874    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"
Nov 02 21:15:14.112912 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:15:14.112872    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:15:36.382689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.382650    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"]
Nov 02 21:15:36.383083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.382929    4528 cpu_manager.go:414] "containerMap: removing container" podUID="89b66e11-7f22-49ad-ae94-235974f17308" containerName="authorino"
Nov 02 21:15:36.383083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.382939    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b66e11-7f22-49ad-ae94-235974f17308" containerName="authorino"
Nov 02 21:15:36.383083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.382988    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b66e11-7f22-49ad-ae94-235974f17308" containerName="authorino"
Nov 02 21:15:36.386890 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.386872    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.480062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.480032    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"]
Nov 02 21:15:36.546347 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.546313    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.546511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.546370    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.546511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.546392    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.546511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.546498    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c167e9c0-18fb-447d-b5ce-febde46c229f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.546610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.546536    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wvm\" (UniqueName: \"kubernetes.io/projected/c167e9c0-18fb-447d-b5ce-febde46c229f-kube-api-access-q2wvm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.546610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.546560    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.546610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.546605    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c167e9c0-18fb-447d-b5ce-febde46c229f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.647292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.647213    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.647292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.647266    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.647292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.647281    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.647561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.647305    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c167e9c0-18fb-447d-b5ce-febde46c229f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.647561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.647414    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wvm\" (UniqueName: \"kubernetes.io/projected/c167e9c0-18fb-447d-b5ce-febde46c229f-kube-api-access-q2wvm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.647561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.647453    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.647561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.647487    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c167e9c0-18fb-447d-b5ce-febde46c229f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.648239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.648213    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.649707 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.649687    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c167e9c0-18fb-447d-b5ce-febde46c229f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.649805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.649781    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.650139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.650096    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c167e9c0-18fb-447d-b5ce-febde46c229f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.650181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.650104    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.707025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.706999    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wvm\" (UniqueName: \"kubernetes.io/projected/c167e9c0-18fb-447d-b5ce-febde46c229f-kube-api-access-q2wvm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.707149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.707084    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c167e9c0-18fb-447d-b5ce-febde46c229f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n5c5h\" (UID: \"c167e9c0-18fb-447d-b5ce-febde46c229f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:36.998553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:36.998477    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:37.410487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:37.410410    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"]
Nov 02 21:15:37.413086 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:15:37.413053    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc167e9c0_18fb_447d_b5ce_febde46c229f.slice/crio-592e1623795beb6513192db8665f196572b6018890f820e2e64b5e7188920ffa WatchSource:0}: Error finding container 592e1623795beb6513192db8665f196572b6018890f820e2e64b5e7188920ffa: Status 404 returned error can't find the container with id 592e1623795beb6513192db8665f196572b6018890f820e2e64b5e7188920ffa
Nov 02 21:15:37.415302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:37.415271    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:15:37.415388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:37.415332    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:15:37.455119 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:37.455084    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h" event={"ID":"c167e9c0-18fb-447d-b5ce-febde46c229f","Type":"ContainerStarted","Data":"592e1623795beb6513192db8665f196572b6018890f820e2e64b5e7188920ffa"}
Nov 02 21:15:38.460433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:38.460397    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h" event={"ID":"c167e9c0-18fb-447d-b5ce-febde46c229f","Type":"ContainerStarted","Data":"530f780f6f882158a06d4dd1e0a4f63fa69ceec4d54882330915aba697a026d1"}
Nov 02 21:15:38.460879 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:38.460527    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:38.462479 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:38.462452    4528 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-n5c5h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:15:38.462625 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:38.462501    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h" podUID="c167e9c0-18fb-447d-b5ce-febde46c229f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:15:38.652660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:38.652606    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h" podStartSLOduration=2.652590848 podStartE2EDuration="2.652590848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:38.61463107 +0000 UTC m=+425.878167389" watchObservedRunningTime="2026-11-02 21:15:38.652590848 +0000 UTC m=+425.916127164"
Nov 02 21:15:39.464742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:39.464715    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n5c5h"
Nov 02 21:15:39.674865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:39.674829    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"]
Nov 02 21:15:39.675155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:39.675090    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v" podUID="5f5c8a40-b73c-4379-aba6-869ed93e4384" containerName="discovery" containerID="cri-o://cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d" gracePeriod=30
Nov 02 21:15:39.911039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:39.911014    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:15:40.079241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.079211    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-dns-cert\") pod \"5f5c8a40-b73c-4379-aba6-869ed93e4384\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") "
Nov 02 21:15:40.079406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.079253    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-ca-configmap\") pod \"5f5c8a40-b73c-4379-aba6-869ed93e4384\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") "
Nov 02 21:15:40.079406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.079290    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-cacerts\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-cacerts\") pod \"5f5c8a40-b73c-4379-aba6-869ed93e4384\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") "
Nov 02 21:15:40.079502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.079402    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5c8a40-b73c-4379-aba6-869ed93e4384-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5c8a40-b73c-4379-aba6-869ed93e4384-local-certs\") pod \"5f5c8a40-b73c-4379-aba6-869ed93e4384\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") "
Nov 02 21:15:40.079502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.079448    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-kube-api-access-dz28m\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-kube-api-access-dz28m\") pod \"5f5c8a40-b73c-4379-aba6-869ed93e4384\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") "
Nov 02 21:15:40.079604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.079506    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-kubeconfig\") pod \"5f5c8a40-b73c-4379-aba6-869ed93e4384\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") "
Nov 02 21:15:40.079604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.079545    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-token\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-token\") pod \"5f5c8a40-b73c-4379-aba6-869ed93e4384\" (UID: \"5f5c8a40-b73c-4379-aba6-869ed93e4384\") "
Nov 02 21:15:40.079709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.079669    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-ca-configmap" pod "5f5c8a40-b73c-4379-aba6-869ed93e4384" (UID: "5f5c8a40-b73c-4379-aba6-869ed93e4384"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:15:40.079821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.079802    4528 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-ca-configmap\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:40.081646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.081614    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-dns-cert" pod "5f5c8a40-b73c-4379-aba6-869ed93e4384" (UID: "5f5c8a40-b73c-4379-aba6-869ed93e4384"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:15:40.081770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.081689    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-cacerts" pod "5f5c8a40-b73c-4379-aba6-869ed93e4384" (UID: "5f5c8a40-b73c-4379-aba6-869ed93e4384"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:15:40.081882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.081857    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-kube-api-access-dz28m" pod "5f5c8a40-b73c-4379-aba6-869ed93e4384" (UID: "5f5c8a40-b73c-4379-aba6-869ed93e4384"). InnerVolumeSpecName "kube-api-access-dz28m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:15:40.081991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.081891    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5c8a40-b73c-4379-aba6-869ed93e4384-local-certs" pod "5f5c8a40-b73c-4379-aba6-869ed93e4384" (UID: "5f5c8a40-b73c-4379-aba6-869ed93e4384"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:15:40.081991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.081937    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-kubeconfig" pod "5f5c8a40-b73c-4379-aba6-869ed93e4384" (UID: "5f5c8a40-b73c-4379-aba6-869ed93e4384"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:15:40.082460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.082439    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-token" pod "5f5c8a40-b73c-4379-aba6-869ed93e4384" (UID: "5f5c8a40-b73c-4379-aba6-869ed93e4384"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:15:40.180277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.180248    4528 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-cacerts\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:40.180277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.180273    4528 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5c8a40-b73c-4379-aba6-869ed93e4384-local-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:40.180277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.180285    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz28m\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-kube-api-access-dz28m\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:40.180484 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.180295    4528 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-kubeconfig\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:40.180484 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.180303    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:40.180484 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.180313    4528 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5f5c8a40-b73c-4379-aba6-869ed93e4384-istio-csr-dns-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:15:40.469514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.469433    4528 generic.go:356] "Generic (PLEG): container finished" podID="5f5c8a40-b73c-4379-aba6-869ed93e4384" containerID="cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d" exitCode=0
Nov 02 21:15:40.469514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.469481    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"
Nov 02 21:15:40.469971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.469515    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v" event={"ID":"5f5c8a40-b73c-4379-aba6-869ed93e4384","Type":"ContainerDied","Data":"cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d"}
Nov 02 21:15:40.469971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.469562    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v" event={"ID":"5f5c8a40-b73c-4379-aba6-869ed93e4384","Type":"ContainerDied","Data":"cc3ad0dd339f7541dd7782f38cb9bacb1260afcf8acb9ed54f6fb4a5ba878a2c"}
Nov 02 21:15:40.469971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.469582    4528 scope.go:122] "RemoveContainer" containerID="cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d"
Nov 02 21:15:40.478576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.478563    4528 scope.go:122] "RemoveContainer" containerID="cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d"
Nov 02 21:15:40.478804 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:15:40.478786    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d\": container with ID starting with cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d not found: ID does not exist" containerID="cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d"
Nov 02 21:15:40.478855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.478812    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d"} err="failed to get container status \"cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d\": rpc error: code = NotFound desc = could not find container \"cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d\": container with ID starting with cfd1dd0125326e5d8fcc3b21a3014d3970c373c7dcbe88c12ad54a07d67e0f4d not found: ID does not exist"
Nov 02 21:15:40.622236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.622207    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"]
Nov 02 21:15:40.668475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.668453    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z4z9v"]
Nov 02 21:15:40.911265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:40.911233    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5c8a40-b73c-4379-aba6-869ed93e4384" path="/var/lib/kubelet/pods/5f5c8a40-b73c-4379-aba6-869ed93e4384/volumes"
Nov 02 21:15:42.882529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.882491    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9xjsn"]
Nov 02 21:15:42.882930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.882779    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5f5c8a40-b73c-4379-aba6-869ed93e4384" containerName="discovery"
Nov 02 21:15:42.882930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.882789    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5c8a40-b73c-4379-aba6-869ed93e4384" containerName="discovery"
Nov 02 21:15:42.882930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.882835    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5c8a40-b73c-4379-aba6-869ed93e4384" containerName="discovery"
Nov 02 21:15:42.886944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.886923    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:15:42.983762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.983737    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 21:15:42.984263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.984242    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 21:15:42.984388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.984339    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 21:15:42.984429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.984387    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-5sp5r\""
Nov 02 21:15:42.984943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:42.984924    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9xjsn"]
Nov 02 21:15:43.002697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.002672    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4be9c715-baba-45d6-8cd9-19d679e10c2a-cert\") pod \"kserve-controller-manager-776cc66d47-9xjsn\" (UID: \"4be9c715-baba-45d6-8cd9-19d679e10c2a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:15:43.002825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.002715    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nlwg\" (UniqueName: \"kubernetes.io/projected/4be9c715-baba-45d6-8cd9-19d679e10c2a-kube-api-access-2nlwg\") pod \"kserve-controller-manager-776cc66d47-9xjsn\" (UID: \"4be9c715-baba-45d6-8cd9-19d679e10c2a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:15:43.103134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.103096    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4be9c715-baba-45d6-8cd9-19d679e10c2a-cert\") pod \"kserve-controller-manager-776cc66d47-9xjsn\" (UID: \"4be9c715-baba-45d6-8cd9-19d679e10c2a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:15:43.103373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.103151    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nlwg\" (UniqueName: \"kubernetes.io/projected/4be9c715-baba-45d6-8cd9-19d679e10c2a-kube-api-access-2nlwg\") pod \"kserve-controller-manager-776cc66d47-9xjsn\" (UID: \"4be9c715-baba-45d6-8cd9-19d679e10c2a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:15:43.106647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.106621    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4be9c715-baba-45d6-8cd9-19d679e10c2a-cert\") pod \"kserve-controller-manager-776cc66d47-9xjsn\" (UID: \"4be9c715-baba-45d6-8cd9-19d679e10c2a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:15:43.117855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.117831    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rgnpj"]
Nov 02 21:15:43.121088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.121071    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:15:43.156703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.156646    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nlwg\" (UniqueName: \"kubernetes.io/projected/4be9c715-baba-45d6-8cd9-19d679e10c2a-kube-api-access-2nlwg\") pod \"kserve-controller-manager-776cc66d47-9xjsn\" (UID: \"4be9c715-baba-45d6-8cd9-19d679e10c2a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:15:43.169874 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.169851    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-f6dhd\""
Nov 02 21:15:43.169979 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.169927    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 02 21:15:43.174576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.174547    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rgnpj"]
Nov 02 21:15:43.199193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.199175    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:15:43.304756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.304726    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcmvq\" (UniqueName: \"kubernetes.io/projected/cb32ed85-0c0b-4b06-95c0-8cf300e337f5-kube-api-access-tcmvq\") pod \"model-serving-api-75bd89fdfc-rgnpj\" (UID: \"cb32ed85-0c0b-4b06-95c0-8cf300e337f5\") " pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:15:43.304922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.304849    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb32ed85-0c0b-4b06-95c0-8cf300e337f5-tls-certs\") pod \"model-serving-api-75bd89fdfc-rgnpj\" (UID: \"cb32ed85-0c0b-4b06-95c0-8cf300e337f5\") " pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:15:43.405353 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.405317    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb32ed85-0c0b-4b06-95c0-8cf300e337f5-tls-certs\") pod \"model-serving-api-75bd89fdfc-rgnpj\" (UID: \"cb32ed85-0c0b-4b06-95c0-8cf300e337f5\") " pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:15:43.405545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.405369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcmvq\" (UniqueName: \"kubernetes.io/projected/cb32ed85-0c0b-4b06-95c0-8cf300e337f5-kube-api-access-tcmvq\") pod \"model-serving-api-75bd89fdfc-rgnpj\" (UID: \"cb32ed85-0c0b-4b06-95c0-8cf300e337f5\") " pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:15:43.407720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.407664    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb32ed85-0c0b-4b06-95c0-8cf300e337f5-tls-certs\") pod \"model-serving-api-75bd89fdfc-rgnpj\" (UID: \"cb32ed85-0c0b-4b06-95c0-8cf300e337f5\") " pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:15:43.458516 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.458491    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcmvq\" (UniqueName: \"kubernetes.io/projected/cb32ed85-0c0b-4b06-95c0-8cf300e337f5-kube-api-access-tcmvq\") pod \"model-serving-api-75bd89fdfc-rgnpj\" (UID: \"cb32ed85-0c0b-4b06-95c0-8cf300e337f5\") " pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:15:43.559740 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.559711    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9xjsn"]
Nov 02 21:15:43.562634 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:15:43.562605    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be9c715_baba_45d6_8cd9_19d679e10c2a.slice/crio-a4af86f6fa265e6858bfac9c5b7fe62bbc1d2d480e4c1e0876ef6129771a4f8c WatchSource:0}: Error finding container a4af86f6fa265e6858bfac9c5b7fe62bbc1d2d480e4c1e0876ef6129771a4f8c: Status 404 returned error can't find the container with id a4af86f6fa265e6858bfac9c5b7fe62bbc1d2d480e4c1e0876ef6129771a4f8c
Nov 02 21:15:43.732703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:43.732632    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:15:44.094338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:44.094313    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-rgnpj"]
Nov 02 21:15:44.096539 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:15:44.096513    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb32ed85_0c0b_4b06_95c0_8cf300e337f5.slice/crio-fc3706a8ddfee6e3217fdbe01c43f7c88f7b48cbfa982c2acffc748fbe6e54a8 WatchSource:0}: Error finding container fc3706a8ddfee6e3217fdbe01c43f7c88f7b48cbfa982c2acffc748fbe6e54a8: Status 404 returned error can't find the container with id fc3706a8ddfee6e3217fdbe01c43f7c88f7b48cbfa982c2acffc748fbe6e54a8
Nov 02 21:15:44.115058 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:15:44.115020    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:15:44.484866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:44.484769    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj" event={"ID":"cb32ed85-0c0b-4b06-95c0-8cf300e337f5","Type":"ContainerStarted","Data":"fc3706a8ddfee6e3217fdbe01c43f7c88f7b48cbfa982c2acffc748fbe6e54a8"}
Nov 02 21:15:44.486001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:44.485970    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn" event={"ID":"4be9c715-baba-45d6-8cd9-19d679e10c2a","Type":"ContainerStarted","Data":"a4af86f6fa265e6858bfac9c5b7fe62bbc1d2d480e4c1e0876ef6129771a4f8c"}
Nov 02 21:15:52.519796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:52.519764    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj" event={"ID":"cb32ed85-0c0b-4b06-95c0-8cf300e337f5","Type":"ContainerStarted","Data":"ab869cd10f376be51fc4fa32386bfe4042d9616c11efda0efde90c1fbee585d9"}
Nov 02 21:15:52.520320 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:52.519933    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:15:52.627246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:52.627194    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj" podStartSLOduration=1.842928341 podStartE2EDuration="9.627181638s" totalImagesPullingTime="7.784253297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:52.626665827 +0000 UTC m=+439.890202145" watchObservedRunningTime="2026-11-02 21:15:52.627181638 +0000 UTC m=+439.890717955"
Nov 02 21:15:55.531783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:55.531751    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn" event={"ID":"4be9c715-baba-45d6-8cd9-19d679e10c2a","Type":"ContainerStarted","Data":"ee8d4d72b4d18481617589a8b23e8bf77656b69cc65f39cefc65b95b3e47bbf1"}
Nov 02 21:15:55.532187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:55.531989    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:15:55.677798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:15:55.677743    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn" podStartSLOduration=2.032814485 podStartE2EDuration="13.677725557s" totalImagesPullingTime="11.644911072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:55.676104322 +0000 UTC m=+442.939640639" watchObservedRunningTime="2026-11-02 21:15:55.677725557 +0000 UTC m=+442.941261876"
Nov 02 21:16:03.529553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:03.529512    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-rgnpj"
Nov 02 21:16:03.870911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:03.870829    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:16:13.242543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.242502    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xdjrb"]
Nov 02 21:16:13.245833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.245815    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb"
Nov 02 21:16:13.297963 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.297927    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xdjrb"]
Nov 02 21:16:13.338534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.338503    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49l4d\" (UniqueName: \"kubernetes.io/projected/c8459568-ed6d-4e02-95f3-19daa598828a-kube-api-access-49l4d\") pod \"authorino-6f6944bcdd-xdjrb\" (UID: \"c8459568-ed6d-4e02-95f3-19daa598828a\") " pod="kuadrant-system/authorino-6f6944bcdd-xdjrb"
Nov 02 21:16:13.342870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.342846    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tr6zv\""
Nov 02 21:16:13.439190 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.439155    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49l4d\" (UniqueName: \"kubernetes.io/projected/c8459568-ed6d-4e02-95f3-19daa598828a-kube-api-access-49l4d\") pod \"authorino-6f6944bcdd-xdjrb\" (UID: \"c8459568-ed6d-4e02-95f3-19daa598828a\") " pod="kuadrant-system/authorino-6f6944bcdd-xdjrb"
Nov 02 21:16:13.492818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.492747    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49l4d\" (UniqueName: \"kubernetes.io/projected/c8459568-ed6d-4e02-95f3-19daa598828a-kube-api-access-49l4d\") pod \"authorino-6f6944bcdd-xdjrb\" (UID: \"c8459568-ed6d-4e02-95f3-19daa598828a\") " pod="kuadrant-system/authorino-6f6944bcdd-xdjrb"
Nov 02 21:16:13.570723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.570691    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb"
Nov 02 21:16:13.616591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.616557    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xdjrb"]
Nov 02 21:16:13.643136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.642240    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c876c946c-zd7k8"]
Nov 02 21:16:13.646799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.646774    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c876c946c-zd7k8"
Nov 02 21:16:13.696495 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.696461    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c876c946c-zd7k8"]
Nov 02 21:16:13.741440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.741408    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvrr\" (UniqueName: \"kubernetes.io/projected/2cf84e08-7d7f-481d-b572-eb5a36157c72-kube-api-access-gcvrr\") pod \"authorino-5c876c946c-zd7k8\" (UID: \"2cf84e08-7d7f-481d-b572-eb5a36157c72\") " pod="kuadrant-system/authorino-5c876c946c-zd7k8"
Nov 02 21:16:13.842281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.842243    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvrr\" (UniqueName: \"kubernetes.io/projected/2cf84e08-7d7f-481d-b572-eb5a36157c72-kube-api-access-gcvrr\") pod \"authorino-5c876c946c-zd7k8\" (UID: \"2cf84e08-7d7f-481d-b572-eb5a36157c72\") " pod="kuadrant-system/authorino-5c876c946c-zd7k8"
Nov 02 21:16:13.897697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.897663    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvrr\" (UniqueName: \"kubernetes.io/projected/2cf84e08-7d7f-481d-b572-eb5a36157c72-kube-api-access-gcvrr\") pod \"authorino-5c876c946c-zd7k8\" (UID: \"2cf84e08-7d7f-481d-b572-eb5a36157c72\") " pod="kuadrant-system/authorino-5c876c946c-zd7k8"
Nov 02 21:16:13.934425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.934395    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xdjrb"]
Nov 02 21:16:13.937474 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:16:13.937438    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8459568_ed6d_4e02_95f3_19daa598828a.slice/crio-a2c5e5ad579eb594d38c9ae067036d9a72736f46445dcb60c8df4b8ddee053e3 WatchSource:0}: Error finding container a2c5e5ad579eb594d38c9ae067036d9a72736f46445dcb60c8df4b8ddee053e3: Status 404 returned error can't find the container with id a2c5e5ad579eb594d38c9ae067036d9a72736f46445dcb60c8df4b8ddee053e3
Nov 02 21:16:13.958748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.958724    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c876c946c-zd7k8"
Nov 02 21:16:13.998299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:13.998266    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c876c946c-zd7k8"]
Nov 02 21:16:14.113684 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:16:14.113595    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:16:14.320249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:14.320213    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c876c946c-zd7k8"]
Nov 02 21:16:14.323668 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:16:14.323635    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cf84e08_7d7f_481d_b572_eb5a36157c72.slice/crio-76c216bfdc48161c6726b35e2fd497833dcffaff45e65d9c51e678c43433c33c WatchSource:0}: Error finding container 76c216bfdc48161c6726b35e2fd497833dcffaff45e65d9c51e678c43433c33c: Status 404 returned error can't find the container with id 76c216bfdc48161c6726b35e2fd497833dcffaff45e65d9c51e678c43433c33c
Nov 02 21:16:14.602403 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:14.602368    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c876c946c-zd7k8" event={"ID":"2cf84e08-7d7f-481d-b572-eb5a36157c72","Type":"ContainerStarted","Data":"76c216bfdc48161c6726b35e2fd497833dcffaff45e65d9c51e678c43433c33c"}
Nov 02 21:16:14.603432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:14.603409    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb" event={"ID":"c8459568-ed6d-4e02-95f3-19daa598828a","Type":"ContainerStarted","Data":"a2c5e5ad579eb594d38c9ae067036d9a72736f46445dcb60c8df4b8ddee053e3"}
Nov 02 21:16:15.608709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:15.608668    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb" event={"ID":"c8459568-ed6d-4e02-95f3-19daa598828a","Type":"ContainerStarted","Data":"b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1"}
Nov 02 21:16:15.609096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:15.608758    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb" podUID="c8459568-ed6d-4e02-95f3-19daa598828a" containerName="authorino" containerID="cri-o://b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1" gracePeriod=30
Nov 02 21:16:15.713969 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:15.713912    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb" podStartSLOduration=1.101599073 podStartE2EDuration="2.713894611s" totalImagesPullingTime="1.612295538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:16:15.712955924 +0000 UTC m=+462.976492243" watchObservedRunningTime="2026-11-02 21:16:15.713894611 +0000 UTC m=+462.977430936"
Nov 02 21:16:15.855508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:15.855484    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb"
Nov 02 21:16:15.960022 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:15.959983    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8459568-ed6d-4e02-95f3-19daa598828a-kube-api-access-49l4d\" (UniqueName: \"kubernetes.io/projected/c8459568-ed6d-4e02-95f3-19daa598828a-kube-api-access-49l4d\") pod \"c8459568-ed6d-4e02-95f3-19daa598828a\" (UID: \"c8459568-ed6d-4e02-95f3-19daa598828a\") "
Nov 02 21:16:15.962594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:15.962548    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8459568-ed6d-4e02-95f3-19daa598828a-kube-api-access-49l4d" pod "c8459568-ed6d-4e02-95f3-19daa598828a" (UID: "c8459568-ed6d-4e02-95f3-19daa598828a"). InnerVolumeSpecName "kube-api-access-49l4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:16:16.061426 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.061386    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49l4d\" (UniqueName: \"kubernetes.io/projected/c8459568-ed6d-4e02-95f3-19daa598828a-kube-api-access-49l4d\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:16.613932 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.613880    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c876c946c-zd7k8" event={"ID":"2cf84e08-7d7f-481d-b572-eb5a36157c72","Type":"ContainerStarted","Data":"8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2"}
Nov 02 21:16:16.615172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.613933    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5c876c946c-zd7k8" podUID="2cf84e08-7d7f-481d-b572-eb5a36157c72" containerName="authorino" containerID="cri-o://8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2" gracePeriod=30
Nov 02 21:16:16.615524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.615353    4528 generic.go:356] "Generic (PLEG): container finished" podID="c8459568-ed6d-4e02-95f3-19daa598828a" containerID="b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1" exitCode=2
Nov 02 21:16:16.615524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.615414    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb"
Nov 02 21:16:16.615524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.615448    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb" event={"ID":"c8459568-ed6d-4e02-95f3-19daa598828a","Type":"ContainerDied","Data":"b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1"}
Nov 02 21:16:16.615524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.615498    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-xdjrb" event={"ID":"c8459568-ed6d-4e02-95f3-19daa598828a","Type":"ContainerDied","Data":"a2c5e5ad579eb594d38c9ae067036d9a72736f46445dcb60c8df4b8ddee053e3"}
Nov 02 21:16:16.615524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.615521    4528 scope.go:122] "RemoveContainer" containerID="b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1"
Nov 02 21:16:16.623816 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.623799    4528 scope.go:122] "RemoveContainer" containerID="b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1"
Nov 02 21:16:16.624096 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:16:16.624077    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1\": container with ID starting with b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1 not found: ID does not exist" containerID="b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1"
Nov 02 21:16:16.624275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.624121    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1"} err="failed to get container status \"b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1\": rpc error: code = NotFound desc = could not find container \"b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1\": container with ID starting with b2ddf2f98e8361316e43932f4847d006e2e78c3ec27ffe447d87ea329a41f4f1 not found: ID does not exist"
Nov 02 21:16:16.718913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.718852    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5c876c946c-zd7k8" podStartSLOduration=2.301202457 podStartE2EDuration="3.718837513s" totalImagesPullingTime="1.417635056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:16:16.71850771 +0000 UTC m=+463.982044028" watchObservedRunningTime="2026-11-02 21:16:16.718837513 +0000 UTC m=+463.982373830"
Nov 02 21:16:16.823027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.822925    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xdjrb"]
Nov 02 21:16:16.826571 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.826543    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xdjrb"]
Nov 02 21:16:16.857908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.857885    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c876c946c-zd7k8"
Nov 02 21:16:16.910896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.910866    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8459568-ed6d-4e02-95f3-19daa598828a" path="/var/lib/kubelet/pods/c8459568-ed6d-4e02-95f3-19daa598828a/volumes"
Nov 02 21:16:16.967531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.967503    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf84e08-7d7f-481d-b572-eb5a36157c72-kube-api-access-gcvrr\" (UniqueName: \"kubernetes.io/projected/2cf84e08-7d7f-481d-b572-eb5a36157c72-kube-api-access-gcvrr\") pod \"2cf84e08-7d7f-481d-b572-eb5a36157c72\" (UID: \"2cf84e08-7d7f-481d-b572-eb5a36157c72\") "
Nov 02 21:16:16.970217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:16.970182    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf84e08-7d7f-481d-b572-eb5a36157c72-kube-api-access-gcvrr" pod "2cf84e08-7d7f-481d-b572-eb5a36157c72" (UID: "2cf84e08-7d7f-481d-b572-eb5a36157c72"). InnerVolumeSpecName "kube-api-access-gcvrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:16:17.069372 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.069337    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcvrr\" (UniqueName: \"kubernetes.io/projected/2cf84e08-7d7f-481d-b572-eb5a36157c72-kube-api-access-gcvrr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:16:17.620410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.620375    4528 generic.go:356] "Generic (PLEG): container finished" podID="2cf84e08-7d7f-481d-b572-eb5a36157c72" containerID="8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2" exitCode=0
Nov 02 21:16:17.620889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.620423    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c876c946c-zd7k8"
Nov 02 21:16:17.620889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.620460    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c876c946c-zd7k8" event={"ID":"2cf84e08-7d7f-481d-b572-eb5a36157c72","Type":"ContainerDied","Data":"8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2"}
Nov 02 21:16:17.620889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.620507    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c876c946c-zd7k8" event={"ID":"2cf84e08-7d7f-481d-b572-eb5a36157c72","Type":"ContainerDied","Data":"76c216bfdc48161c6726b35e2fd497833dcffaff45e65d9c51e678c43433c33c"}
Nov 02 21:16:17.620889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.620533    4528 scope.go:122] "RemoveContainer" containerID="8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2"
Nov 02 21:16:17.628504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.628487    4528 scope.go:122] "RemoveContainer" containerID="8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2"
Nov 02 21:16:17.628790 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:16:17.628741    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2\": container with ID starting with 8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2 not found: ID does not exist" containerID="8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2"
Nov 02 21:16:17.628790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.628774    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2"} err="failed to get container status \"8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2\": rpc error: code = NotFound desc = could not find container \"8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2\": container with ID starting with 8b5ef3120c9e79be0f7a8e061c3403f5f3047166b1fcd29e14426845634f4be2 not found: ID does not exist"
Nov 02 21:16:17.771860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.771825    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c876c946c-zd7k8"]
Nov 02 21:16:17.775822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:17.775797    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c876c946c-zd7k8"]
Nov 02 21:16:18.911070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:18.911035    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf84e08-7d7f-481d-b572-eb5a36157c72" path="/var/lib/kubelet/pods/2cf84e08-7d7f-481d-b572-eb5a36157c72/volumes"
Nov 02 21:16:26.541556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:16:26.541518    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9xjsn"
Nov 02 21:16:44.112335 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:16:44.112297    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:17:14.113239 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:17:14.113201    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:17:31.745117 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.745080    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"]
Nov 02 21:17:31.745567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.745484    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c8459568-ed6d-4e02-95f3-19daa598828a" containerName="authorino"
Nov 02 21:17:31.745567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.745503    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8459568-ed6d-4e02-95f3-19daa598828a" containerName="authorino"
Nov 02 21:17:31.745567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.745541    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2cf84e08-7d7f-481d-b572-eb5a36157c72" containerName="authorino"
Nov 02 21:17:31.745567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.745549    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf84e08-7d7f-481d-b572-eb5a36157c72" containerName="authorino"
Nov 02 21:17:31.745706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.745607    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8459568-ed6d-4e02-95f3-19daa598828a" containerName="authorino"
Nov 02 21:17:31.745706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.745617    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf84e08-7d7f-481d-b572-eb5a36157c72" containerName="authorino"
Nov 02 21:17:31.748504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.748487    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:31.800878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.800812    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 21:17:31.801142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.801102    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 21:17:31.847036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.847002    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-4xndc\""
Nov 02 21:17:31.848119 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.848088    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"]
Nov 02 21:17:31.872189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.872162    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvw4\" (UniqueName: \"kubernetes.io/projected/7e1d1c0c-f51a-461d-8d27-4caa553172ca-kube-api-access-swvw4\") pod \"ai-gateway-controller-5d8dddb676-s6wgt\" (UID: \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:31.872338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.872205    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1d1c0c-f51a-461d-8d27-4caa553172ca-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-s6wgt\" (UID: \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:31.973273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.973237    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1d1c0c-f51a-461d-8d27-4caa553172ca-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-s6wgt\" (UID: \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:31.973469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.973365    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvw4\" (UniqueName: \"kubernetes.io/projected/7e1d1c0c-f51a-461d-8d27-4caa553172ca-kube-api-access-swvw4\") pod \"ai-gateway-controller-5d8dddb676-s6wgt\" (UID: \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:31.975656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:31.975634    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1d1c0c-f51a-461d-8d27-4caa553172ca-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-s6wgt\" (UID: \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:32.027024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:32.026991    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvw4\" (UniqueName: \"kubernetes.io/projected/7e1d1c0c-f51a-461d-8d27-4caa553172ca-kube-api-access-swvw4\") pod \"ai-gateway-controller-5d8dddb676-s6wgt\" (UID: \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:32.061007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:32.060975    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:32.419665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:32.419584    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"]
Nov 02 21:17:32.912872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:32.912821    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt" event={"ID":"7e1d1c0c-f51a-461d-8d27-4caa553172ca","Type":"ContainerStarted","Data":"b0c92b67ed8b047a5ed63907db031eee1308c1c4ff80b1d5833f602d97804647"}
Nov 02 21:17:35.614894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:35.614861    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 21:17:35.930325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:35.930231    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt" event={"ID":"7e1d1c0c-f51a-461d-8d27-4caa553172ca","Type":"ContainerStarted","Data":"0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47"}
Nov 02 21:17:35.930502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:35.930366    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:36.047243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:36.047183    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt" podStartSLOduration=1.906621235 podStartE2EDuration="5.047166178s" totalImagesPullingTime="3.140544943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:17:36.045892044 +0000 UTC m=+543.309428387" watchObservedRunningTime="2026-11-02 21:17:36.047166178 +0000 UTC m=+543.310702575"
Nov 02 21:17:39.926447 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:39.926411    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"]
Nov 02 21:17:39.929729 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:39.929702    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:39.982619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:39.982583    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"]
Nov 02 21:17:40.027217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.027182    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 21:17:40.027379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.027276    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tdwwv\""
Nov 02 21:17:40.039417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.039391    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brt9p\" (UniqueName: \"kubernetes.io/projected/60623367-8d1f-47b8-b90c-d7830b8a73b2-kube-api-access-brt9p\") pod \"payload-pre-processing-7b98bb678c-kzlc6\" (UID: \"60623367-8d1f-47b8-b90c-d7830b8a73b2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:40.039558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.039457    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60623367-8d1f-47b8-b90c-d7830b8a73b2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kzlc6\" (UID: \"60623367-8d1f-47b8-b90c-d7830b8a73b2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:40.077418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.077381    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"]
Nov 02 21:17:40.080684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.080662    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:17:40.133820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.133790    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"]
Nov 02 21:17:40.140877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.140850    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60623367-8d1f-47b8-b90c-d7830b8a73b2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kzlc6\" (UID: \"60623367-8d1f-47b8-b90c-d7830b8a73b2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:40.141037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.140916    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brt9p\" (UniqueName: \"kubernetes.io/projected/60623367-8d1f-47b8-b90c-d7830b8a73b2-kube-api-access-brt9p\") pod \"payload-pre-processing-7b98bb678c-kzlc6\" (UID: \"60623367-8d1f-47b8-b90c-d7830b8a73b2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:40.141609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.141589    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60623367-8d1f-47b8-b90c-d7830b8a73b2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kzlc6\" (UID: \"60623367-8d1f-47b8-b90c-d7830b8a73b2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:40.194671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.194603    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brt9p\" (UniqueName: \"kubernetes.io/projected/60623367-8d1f-47b8-b90c-d7830b8a73b2-kube-api-access-brt9p\") pod \"payload-pre-processing-7b98bb678c-kzlc6\" (UID: \"60623367-8d1f-47b8-b90c-d7830b8a73b2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:40.241986 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.241956    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:40.242171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.242137    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8kh\" (UniqueName: \"kubernetes.io/projected/53e705a3-6036-4fdd-800c-90b4eb36f5a8-kube-api-access-kn8kh\") pod \"payload-processing-55fbbbf9dc-pwkmv\" (UID: \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:17:40.242316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.242295    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e705a3-6036-4fdd-800c-90b4eb36f5a8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pwkmv\" (UID: \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:17:40.342822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.342789    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e705a3-6036-4fdd-800c-90b4eb36f5a8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pwkmv\" (UID: \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:17:40.343025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.342830    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8kh\" (UniqueName: \"kubernetes.io/projected/53e705a3-6036-4fdd-800c-90b4eb36f5a8-kube-api-access-kn8kh\") pod \"payload-processing-55fbbbf9dc-pwkmv\" (UID: \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:17:40.343530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.343510    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e705a3-6036-4fdd-800c-90b4eb36f5a8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pwkmv\" (UID: \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:17:40.396071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.396040    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8kh\" (UniqueName: \"kubernetes.io/projected/53e705a3-6036-4fdd-800c-90b4eb36f5a8-kube-api-access-kn8kh\") pod \"payload-processing-55fbbbf9dc-pwkmv\" (UID: \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:17:40.605424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.605394    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"]
Nov 02 21:17:40.607910 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:17:40.607879    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60623367_8d1f_47b8_b90c_d7830b8a73b2.slice/crio-324a5c31039320466de741f9cd07c3d36ed3f9e7cd1903a45932fa97d2e69a70 WatchSource:0}: Error finding container 324a5c31039320466de741f9cd07c3d36ed3f9e7cd1903a45932fa97d2e69a70: Status 404 returned error can't find the container with id 324a5c31039320466de741f9cd07c3d36ed3f9e7cd1903a45932fa97d2e69a70
Nov 02 21:17:40.693644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.693610    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:17:40.954360 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:40.954276    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6" event={"ID":"60623367-8d1f-47b8-b90c-d7830b8a73b2","Type":"ContainerStarted","Data":"324a5c31039320466de741f9cd07c3d36ed3f9e7cd1903a45932fa97d2e69a70"}
Nov 02 21:17:41.056253 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:41.056216    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"]
Nov 02 21:17:41.059965 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:17:41.059938    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e705a3_6036_4fdd_800c_90b4eb36f5a8.slice/crio-f01ecfaaeda10d8a2f803f564a05b051848c84cc55908d2703c416810a383c0f WatchSource:0}: Error finding container f01ecfaaeda10d8a2f803f564a05b051848c84cc55908d2703c416810a383c0f: Status 404 returned error can't find the container with id f01ecfaaeda10d8a2f803f564a05b051848c84cc55908d2703c416810a383c0f
Nov 02 21:17:41.959399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:41.959364    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv" event={"ID":"53e705a3-6036-4fdd-800c-90b4eb36f5a8","Type":"ContainerStarted","Data":"f01ecfaaeda10d8a2f803f564a05b051848c84cc55908d2703c416810a383c0f"}
Nov 02 21:17:43.971232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:43.971148    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6" event={"ID":"60623367-8d1f-47b8-b90c-d7830b8a73b2","Type":"ContainerStarted","Data":"e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f"}
Nov 02 21:17:43.971689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:43.971250    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:44.078817 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:44.078769    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6" podStartSLOduration=2.000259575 podStartE2EDuration="5.078754775s" totalImagesPullingTime="3.0784952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:17:44.077358775 +0000 UTC m=+551.340895095" watchObservedRunningTime="2026-11-02 21:17:44.078754775 +0000 UTC m=+551.342291089"
Nov 02 21:17:44.114580 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:17:44.114550    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:17:44.977524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:44.977482    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv" event={"ID":"53e705a3-6036-4fdd-800c-90b4eb36f5a8","Type":"ContainerStarted","Data":"4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99"}
Nov 02 21:17:44.978012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:44.977730    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:17:45.083577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:45.083529    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv" podStartSLOduration=2.042315689 podStartE2EDuration="5.083517182s" totalImagesPullingTime="3.041201493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:17:45.083240669 +0000 UTC m=+552.346776987" watchObservedRunningTime="2026-11-02 21:17:45.083517182 +0000 UTC m=+552.347053498"
Nov 02 21:17:46.936865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:46.936821    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:17:47.278315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.278280    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-74879c7667-jgdxh"]
Nov 02 21:17:47.281518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.281499    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:17:47.335865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.335835    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74879c7667-jgdxh"]
Nov 02 21:17:47.404769 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.404738    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9f5b\" (UniqueName: \"kubernetes.io/projected/8ddc5d0a-c7a0-412a-ac84-c522dfcba78c-kube-api-access-k9f5b\") pod \"ai-gateway-controller-74879c7667-jgdxh\" (UID: \"8ddc5d0a-c7a0-412a-ac84-c522dfcba78c\") " pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:17:47.404971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.404824    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ddc5d0a-c7a0-412a-ac84-c522dfcba78c-metrics-certs\") pod \"ai-gateway-controller-74879c7667-jgdxh\" (UID: \"8ddc5d0a-c7a0-412a-ac84-c522dfcba78c\") " pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:17:47.506156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.506098    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ddc5d0a-c7a0-412a-ac84-c522dfcba78c-metrics-certs\") pod \"ai-gateway-controller-74879c7667-jgdxh\" (UID: \"8ddc5d0a-c7a0-412a-ac84-c522dfcba78c\") " pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:17:47.506324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.506173    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9f5b\" (UniqueName: \"kubernetes.io/projected/8ddc5d0a-c7a0-412a-ac84-c522dfcba78c-kube-api-access-k9f5b\") pod \"ai-gateway-controller-74879c7667-jgdxh\" (UID: \"8ddc5d0a-c7a0-412a-ac84-c522dfcba78c\") " pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:17:47.508494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.508474    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ddc5d0a-c7a0-412a-ac84-c522dfcba78c-metrics-certs\") pod \"ai-gateway-controller-74879c7667-jgdxh\" (UID: \"8ddc5d0a-c7a0-412a-ac84-c522dfcba78c\") " pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:17:47.559641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.559568    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9f5b\" (UniqueName: \"kubernetes.io/projected/8ddc5d0a-c7a0-412a-ac84-c522dfcba78c-kube-api-access-k9f5b\") pod \"ai-gateway-controller-74879c7667-jgdxh\" (UID: \"8ddc5d0a-c7a0-412a-ac84-c522dfcba78c\") " pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:17:47.593462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.593428    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:17:47.953296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.953270    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74879c7667-jgdxh"]
Nov 02 21:17:47.955501 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:17:47.955472    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddc5d0a_c7a0_412a_ac84_c522dfcba78c.slice/crio-74d159c53939520b1727396e7437f606ce7bd79fe58d92fa20f63beea596af29 WatchSource:0}: Error finding container 74d159c53939520b1727396e7437f606ce7bd79fe58d92fa20f63beea596af29: Status 404 returned error can't find the container with id 74d159c53939520b1727396e7437f606ce7bd79fe58d92fa20f63beea596af29
Nov 02 21:17:47.991212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:47.991178    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh" event={"ID":"8ddc5d0a-c7a0-412a-ac84-c522dfcba78c","Type":"ContainerStarted","Data":"74d159c53939520b1727396e7437f606ce7bd79fe58d92fa20f63beea596af29"}
Nov 02 21:17:48.996497 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:48.996462    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh" event={"ID":"8ddc5d0a-c7a0-412a-ac84-c522dfcba78c","Type":"ContainerStarted","Data":"ca74fcbac8d0af1830d05a5782ebbbc85585202d5e0eee158e1045457775571f"}
Nov 02 21:17:48.996936 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:48.996589    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:17:49.104026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:49.103969    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh" podStartSLOduration=1.198855866 podStartE2EDuration="2.103954147s" totalImagesPullingTime="905.098281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:17:49.103070495 +0000 UTC m=+556.366606811" watchObservedRunningTime="2026-11-02 21:17:49.103954147 +0000 UTC m=+556.367490514"
Nov 02 21:17:54.981319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:54.981285    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 21:17:55.985503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:17:55.985475    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 21:18:00.002989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.002959    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-74879c7667-jgdxh"
Nov 02 21:18:00.157525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.157492    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"]
Nov 02 21:18:00.157986 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.157955    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt" podUID="7e1d1c0c-f51a-461d-8d27-4caa553172ca" containerName="manager" containerID="cri-o://0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47" gracePeriod=30
Nov 02 21:18:00.407416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.407390    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:18:00.417333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.417309    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e1d1c0c-f51a-461d-8d27-4caa553172ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1d1c0c-f51a-461d-8d27-4caa553172ca-metrics-certs\") pod \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\" (UID: \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\") "
Nov 02 21:18:00.417460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.417385    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1d1c0c-f51a-461d-8d27-4caa553172ca-kube-api-access-swvw4\" (UniqueName: \"kubernetes.io/projected/7e1d1c0c-f51a-461d-8d27-4caa553172ca-kube-api-access-swvw4\") pod \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\" (UID: \"7e1d1c0c-f51a-461d-8d27-4caa553172ca\") "
Nov 02 21:18:00.419411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.419381    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1d1c0c-f51a-461d-8d27-4caa553172ca-metrics-certs" pod "7e1d1c0c-f51a-461d-8d27-4caa553172ca" (UID: "7e1d1c0c-f51a-461d-8d27-4caa553172ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:18:00.419501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.419452    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1d1c0c-f51a-461d-8d27-4caa553172ca-kube-api-access-swvw4" pod "7e1d1c0c-f51a-461d-8d27-4caa553172ca" (UID: "7e1d1c0c-f51a-461d-8d27-4caa553172ca"). InnerVolumeSpecName "kube-api-access-swvw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:18:00.518471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.518404    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvw4\" (UniqueName: \"kubernetes.io/projected/7e1d1c0c-f51a-461d-8d27-4caa553172ca-kube-api-access-swvw4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:00.518471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:00.518428    4528 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1d1c0c-f51a-461d-8d27-4caa553172ca-metrics-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:18:01.048176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:01.048139    4528 generic.go:356] "Generic (PLEG): container finished" podID="7e1d1c0c-f51a-461d-8d27-4caa553172ca" containerID="0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47" exitCode=0
Nov 02 21:18:01.048176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:01.048174    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"
Nov 02 21:18:01.048602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:01.048217    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt" event={"ID":"7e1d1c0c-f51a-461d-8d27-4caa553172ca","Type":"ContainerDied","Data":"0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47"}
Nov 02 21:18:01.048602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:01.048267    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-s6wgt" event={"ID":"7e1d1c0c-f51a-461d-8d27-4caa553172ca","Type":"ContainerDied","Data":"b0c92b67ed8b047a5ed63907db031eee1308c1c4ff80b1d5833f602d97804647"}
Nov 02 21:18:01.048602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:01.048288    4528 scope.go:122] "RemoveContainer" containerID="0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47"
Nov 02 21:18:01.056041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:01.056024    4528 scope.go:122] "RemoveContainer" containerID="0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47"
Nov 02 21:18:01.056303 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:18:01.056285    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47\": container with ID starting with 0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47 not found: ID does not exist" containerID="0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47"
Nov 02 21:18:01.056374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:01.056313    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47"} err="failed to get container status \"0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47\": rpc error: code = NotFound desc = could not find container \"0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47\": container with ID starting with 0af110131f9441e08cd453a66877006f9b771af289f1c6404b8f7825dca20c47 not found: ID does not exist"
Nov 02 21:18:01.203285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:01.203198    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"]
Nov 02 21:18:01.249716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:01.249685    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-s6wgt"]
Nov 02 21:18:02.911517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:02.911476    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1d1c0c-f51a-461d-8d27-4caa553172ca" path="/var/lib/kubelet/pods/7e1d1c0c-f51a-461d-8d27-4caa553172ca/volumes"
Nov 02 21:18:14.112502 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:18:14.112467    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:18:16.815461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:16.815427    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:18:22.922688 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:22.922653    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"]
Nov 02 21:18:22.923068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:22.922966    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7e1d1c0c-f51a-461d-8d27-4caa553172ca" containerName="manager"
Nov 02 21:18:22.923068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:22.922975    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1d1c0c-f51a-461d-8d27-4caa553172ca" containerName="manager"
Nov 02 21:18:22.923068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:22.923026    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1d1c0c-f51a-461d-8d27-4caa553172ca" containerName="manager"
Nov 02 21:18:22.926047 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:22.926023    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:22.974667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:22.974636    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 21:18:22.974845 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:22.974688    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 21:18:23.009927 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.009897    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31358659-6618-4da0-90bd-1d69aefa7ffd-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.010084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.009954    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.010084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.010042    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.010084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.010069    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.010216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.010095    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.010281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.010263    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2fc4\" (UniqueName: \"kubernetes.io/projected/31358659-6618-4da0-90bd-1d69aefa7ffd-kube-api-access-w2fc4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.021227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.021199    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4dfx7\""
Nov 02 21:18:23.022463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.022443    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"]
Nov 02 21:18:23.074851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.074815    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 21:18:23.111296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.111267    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.111463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.111319    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2fc4\" (UniqueName: \"kubernetes.io/projected/31358659-6618-4da0-90bd-1d69aefa7ffd-kube-api-access-w2fc4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.111463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.111342    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31358659-6618-4da0-90bd-1d69aefa7ffd-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.111463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.111386    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.111463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.111409    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.111705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.111613    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.111822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.111795    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.111892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.111834    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.111929 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.111914    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.113565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.113549    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31358659-6618-4da0-90bd-1d69aefa7ffd-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.113717 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.113700    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31358659-6618-4da0-90bd-1d69aefa7ffd-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.164156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.164127    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2fc4\" (UniqueName: \"kubernetes.io/projected/31358659-6618-4da0-90bd-1d69aefa7ffd-kube-api-access-w2fc4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zgd28\" (UID: \"31358659-6618-4da0-90bd-1d69aefa7ffd\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.238424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.238342    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:23.601791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:23.601759    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"]
Nov 02 21:18:23.605164 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:18:23.605129    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31358659_6618_4da0_90bd_1d69aefa7ffd.slice/crio-cb19759adda6580f5b54d4b130b5ecdfa51656b292295c1ab91a922e9b1a8ed3 WatchSource:0}: Error finding container cb19759adda6580f5b54d4b130b5ecdfa51656b292295c1ab91a922e9b1a8ed3: Status 404 returned error can't find the container with id cb19759adda6580f5b54d4b130b5ecdfa51656b292295c1ab91a922e9b1a8ed3
Nov 02 21:18:24.138942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:24.138911    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28" event={"ID":"31358659-6618-4da0-90bd-1d69aefa7ffd","Type":"ContainerStarted","Data":"cb19759adda6580f5b54d4b130b5ecdfa51656b292295c1ab91a922e9b1a8ed3"}
Nov 02 21:18:26.149135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:26.149091    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28" event={"ID":"31358659-6618-4da0-90bd-1d69aefa7ffd","Type":"ContainerStarted","Data":"1761d80c5867b838d569cd387c6a66eb966b8fb132dbe6b969892d00cc2ee719"}
Nov 02 21:18:26.239288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:26.239247    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:26.256754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:26.256709    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28" podStartSLOduration=2.712230252 podStartE2EDuration="4.25669569s" totalImagesPullingTime="1.544465438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:26.256162117 +0000 UTC m=+593.519698435" watchObservedRunningTime="2026-11-02 21:18:26.25669569 +0000 UTC m=+593.520232007"
Nov 02 21:18:26.995366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:26.995335    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"]
Nov 02 21:18:26.998666 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:26.998647    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.047606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.047572    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 21:18:27.094976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.094944    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"]
Nov 02 21:18:27.095946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.095924    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:18:27.149128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.149083    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.149618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.149145    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.149618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.149266    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.149618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.149356    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.149618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.149419    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf97d442-ef72-4399-a586-88274dfbd464-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.149618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.149521    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wm9\" (UniqueName: \"kubernetes.io/projected/bf97d442-ef72-4399-a586-88274dfbd464-kube-api-access-t7wm9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.250714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.250628    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.250714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.250695    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.250952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.250728    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.250952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.250748    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf97d442-ef72-4399-a586-88274dfbd464-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.250952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.250801    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wm9\" (UniqueName: \"kubernetes.io/projected/bf97d442-ef72-4399-a586-88274dfbd464-kube-api-access-t7wm9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.250952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.250850    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.251490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.251463    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.251615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.251524    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.251615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.251558    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.253418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.253396    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf97d442-ef72-4399-a586-88274dfbd464-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.253574 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.253557    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf97d442-ef72-4399-a586-88274dfbd464-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.303644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.303608    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wm9\" (UniqueName: \"kubernetes.io/projected/bf97d442-ef72-4399-a586-88274dfbd464-kube-api-access-t7wm9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rmwm5\" (UID: \"bf97d442-ef72-4399-a586-88274dfbd464\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.310922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.310896    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:27.676476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:27.676448    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"]
Nov 02 21:18:27.678918 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:18:27.678889    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf97d442_ef72_4399_a586_88274dfbd464.slice/crio-400e7363a70aaea6462018e999741203bcf5b8bf0e75107edd943403ab940792 WatchSource:0}: Error finding container 400e7363a70aaea6462018e999741203bcf5b8bf0e75107edd943403ab940792: Status 404 returned error can't find the container with id 400e7363a70aaea6462018e999741203bcf5b8bf0e75107edd943403ab940792
Nov 02 21:18:28.157649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:28.157616    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5" event={"ID":"bf97d442-ef72-4399-a586-88274dfbd464","Type":"ContainerStarted","Data":"400e7363a70aaea6462018e999741203bcf5b8bf0e75107edd943403ab940792"}
Nov 02 21:18:29.162764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:29.162728    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5" event={"ID":"bf97d442-ef72-4399-a586-88274dfbd464","Type":"ContainerStarted","Data":"123d712992a6c5472c8892eee4341524cfc71e10dfde15641037e1d363c9fbef"}
Nov 02 21:18:29.270344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:29.270292    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5" podStartSLOduration=2.643080399 podStartE2EDuration="3.270278783s" totalImagesPullingTime="627.198384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:29.269706585 +0000 UTC m=+596.533242904" watchObservedRunningTime="2026-11-02 21:18:29.270278783 +0000 UTC m=+596.533815099"
Nov 02 21:18:29.311196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:29.311163    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:33.239583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:33.239491    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:33.251724 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:33.251692    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:34.193881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:34.193844    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zgd28"
Nov 02 21:18:35.021344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.021307    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mqnnt"]
Nov 02 21:18:35.027104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.027082    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.077078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.077043    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mqnnt"]
Nov 02 21:18:35.117770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.117738    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec65c14-0c66-48d0-98ad-c190b3ed0e38-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.117934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.117782    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f6wm\" (UniqueName: \"kubernetes.io/projected/dec65c14-0c66-48d0-98ad-c190b3ed0e38-kube-api-access-5f6wm\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.117934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.117812    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.117934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.117868    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-home\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.118037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.117944    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.118037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.117990    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.121591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.121575    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 21:18:35.218554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.218521    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.218554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.218553    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-home\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.218785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.218588    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.218785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.218620    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.218785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.218644    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec65c14-0c66-48d0-98ad-c190b3ed0e38-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.218785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.218663    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f6wm\" (UniqueName: \"kubernetes.io/projected/dec65c14-0c66-48d0-98ad-c190b3ed0e38-kube-api-access-5f6wm\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.219025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.219003    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-home\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.219080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.219048    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.219080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.219068    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.220736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.220709    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec65c14-0c66-48d0-98ad-c190b3ed0e38-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.221036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.221017    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec65c14-0c66-48d0-98ad-c190b3ed0e38-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.272035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.271968    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f6wm\" (UniqueName: \"kubernetes.io/projected/dec65c14-0c66-48d0-98ad-c190b3ed0e38-kube-api-access-5f6wm\") pod \"echo-simulator-kserve-64b6f8f96c-mqnnt\" (UID: \"dec65c14-0c66-48d0-98ad-c190b3ed0e38\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.339307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.339275    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:35.702439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:35.702370    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mqnnt"]
Nov 02 21:18:35.705252 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:18:35.705225    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec65c14_0c66_48d0_98ad_c190b3ed0e38.slice/crio-794cd64b5bb11fdafb5261947e02b1d83addcdc44c4a53f79ac186dcc8de36d0 WatchSource:0}: Error finding container 794cd64b5bb11fdafb5261947e02b1d83addcdc44c4a53f79ac186dcc8de36d0: Status 404 returned error can't find the container with id 794cd64b5bb11fdafb5261947e02b1d83addcdc44c4a53f79ac186dcc8de36d0
Nov 02 21:18:36.192774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:36.192741    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt" event={"ID":"dec65c14-0c66-48d0-98ad-c190b3ed0e38","Type":"ContainerStarted","Data":"794cd64b5bb11fdafb5261947e02b1d83addcdc44c4a53f79ac186dcc8de36d0"}
Nov 02 21:18:37.311186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:37.311151    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:37.326037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:37.326009    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:38.212573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:38.212541    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rmwm5"
Nov 02 21:18:38.885725 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:38.885692    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:18:39.207385 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:39.207283    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt" event={"ID":"dec65c14-0c66-48d0-98ad-c190b3ed0e38","Type":"ContainerStarted","Data":"dad9c4d4074fd980b83b90db503e5772a5638e1e032d080196a3ba4d0e70b926"}
Nov 02 21:18:39.315410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:39.315349    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt" podStartSLOduration=2.7661213609999997 podStartE2EDuration="5.315335213s" totalImagesPullingTime="2.549213852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:39.313800107 +0000 UTC m=+606.577336428" watchObservedRunningTime="2026-11-02 21:18:39.315335213 +0000 UTC m=+606.578871530"
Nov 02 21:18:39.339509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:39.339474    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:40.587798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.587767    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"]
Nov 02 21:18:40.591198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.591173    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.640035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.639998    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 21:18:40.687617 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.687584    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"]
Nov 02 21:18:40.776695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.776664    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fa4184-1b2b-49ca-8517-06957cad7927-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.776695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.776700    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.776920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.776721    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.776920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.776780    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8s64\" (UniqueName: \"kubernetes.io/projected/92fa4184-1b2b-49ca-8517-06957cad7927-kube-api-access-x8s64\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.776920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.776840    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.776920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.776863    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.877915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.877824    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fa4184-1b2b-49ca-8517-06957cad7927-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.877915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.877863    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.877915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.877883    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.877915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.877904    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8s64\" (UniqueName: \"kubernetes.io/projected/92fa4184-1b2b-49ca-8517-06957cad7927-kube-api-access-x8s64\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.878288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.877959    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.878288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.877991    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.878395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.878352    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.878440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.878410    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.878515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.878496    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.880223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.880196    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fa4184-1b2b-49ca-8517-06957cad7927-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.880412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.880394    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fa4184-1b2b-49ca-8517-06957cad7927-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:40.931476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:40.931449    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8s64\" (UniqueName: \"kubernetes.io/projected/92fa4184-1b2b-49ca-8517-06957cad7927-kube-api-access-x8s64\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dstck\" (UID: \"92fa4184-1b2b-49ca-8517-06957cad7927\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:41.204413 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:41.204329    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:41.568508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:41.568485    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"]
Nov 02 21:18:41.570623 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:18:41.570597    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fa4184_1b2b_49ca_8517_06957cad7927.slice/crio-c27557f8e0be8de4c59cd136f143a33f25f7149297c47592cb247079a2c63c6f WatchSource:0}: Error finding container c27557f8e0be8de4c59cd136f143a33f25f7149297c47592cb247079a2c63c6f: Status 404 returned error can't find the container with id c27557f8e0be8de4c59cd136f143a33f25f7149297c47592cb247079a2c63c6f
Nov 02 21:18:42.222465 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:42.222432    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck" event={"ID":"92fa4184-1b2b-49ca-8517-06957cad7927","Type":"ContainerStarted","Data":"c27557f8e0be8de4c59cd136f143a33f25f7149297c47592cb247079a2c63c6f"}
Nov 02 21:18:43.227982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:43.227948    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck" event={"ID":"92fa4184-1b2b-49ca-8517-06957cad7927","Type":"ContainerStarted","Data":"db8dc9243d0cafb61b2c0d6bd56cf4f7467ce7ae8f5b985d6782d215a688135c"}
Nov 02 21:18:43.335318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:43.335267    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck" podStartSLOduration=2.668768975 podStartE2EDuration="3.335249406s" totalImagesPullingTime="666.480431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:43.333918995 +0000 UTC m=+610.597455300" watchObservedRunningTime="2026-11-02 21:18:43.335249406 +0000 UTC m=+610.598785724"
Nov 02 21:18:44.112194 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:18:44.112152    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:18:44.204922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.204885    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:44.862836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.862800    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"]
Nov 02 21:18:44.866244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.866223    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.913776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.913744    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.913950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.913784    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.914041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.914020    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.914104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.914058    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447bq\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-kube-api-access-447bq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.914230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.914204    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.914296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.914281    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771361df-3dd7-4787-bf88-53a214521356-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.914428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.914409    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.914486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.914443    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771361df-3dd7-4787-bf88-53a214521356-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.914547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.914525    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:44.914913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.914894    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4lw9\""
Nov 02 21:18:44.963079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:44.963040    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"]
Nov 02 21:18:45.015657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.015627    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447bq\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-kube-api-access-447bq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.015843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.015675    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.015843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.015697    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771361df-3dd7-4787-bf88-53a214521356-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.015843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.015743    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.015843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.015761    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771361df-3dd7-4787-bf88-53a214521356-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.015843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.015778    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.015843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.015803    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.016220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.015857    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.016220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.015919    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.016220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.016086    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.016220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.016156    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.016432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.016394    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.016432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.016424    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.016599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.016578    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771361df-3dd7-4787-bf88-53a214521356-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.018611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.018587    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.018714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.018700    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771361df-3dd7-4787-bf88-53a214521356-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.068381 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.068350    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.068381 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.068368    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-447bq\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-kube-api-access-447bq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6qkd9\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.181340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.181248    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:45.339652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.339619    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:45.352305 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.352279    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:45.614220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:45.614191    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"]
Nov 02 21:18:45.616544 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:18:45.616512    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771361df_3dd7_4787_bf88_53a214521356.slice/crio-b3b7d02f7880853ec52c19f213968b1a52d3dcaffe5247d6eb9b6f3751147801 WatchSource:0}: Error finding container b3b7d02f7880853ec52c19f213968b1a52d3dcaffe5247d6eb9b6f3751147801: Status 404 returned error can't find the container with id b3b7d02f7880853ec52c19f213968b1a52d3dcaffe5247d6eb9b6f3751147801
Nov 02 21:18:46.244854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:46.244823    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" event={"ID":"771361df-3dd7-4787-bf88-53a214521356","Type":"ContainerStarted","Data":"b3b7d02f7880853ec52c19f213968b1a52d3dcaffe5247d6eb9b6f3751147801"}
Nov 02 21:18:46.255905 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:46.255878    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mqnnt"
Nov 02 21:18:49.386461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:49.386427    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:18:49.386881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:49.386506    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:18:49.386881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:49.386550    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:18:50.264183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:50.264149    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" event={"ID":"771361df-3dd7-4787-bf88-53a214521356","Type":"ContainerStarted","Data":"2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc"}
Nov 02 21:18:50.473416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:50.473359    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podStartSLOduration=2.706389714 podStartE2EDuration="6.473345139s" totalImagesPullingTime="3.766955425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:50.426531578 +0000 UTC m=+617.690067894" watchObservedRunningTime="2026-11-02 21:18:50.473345139 +0000 UTC m=+617.736881457"
Nov 02 21:18:51.182208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:51.182154    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:51.183871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:51.183842    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6qkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 21:18:51.184009 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:51.183902    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 21:18:51.216568 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:51.211054    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:51.235189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:51.235150    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:51.279580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:51.279553    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dstck"
Nov 02 21:18:52.182309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:52.182274    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6qkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 21:18:52.182703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:52.182326    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 21:18:53.181728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:53.181692    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6qkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 21:18:53.181905 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:53.181754    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 21:18:54.182226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:54.182189    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6qkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 21:18:54.182610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:54.182257    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 21:18:55.182102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:55.182070    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6qkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 21:18:55.182312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:55.182156    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 21:18:56.182016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:56.181976    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6qkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 21:18:56.182232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:56.182043    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 21:18:56.875546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:56.875510    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:18:57.567405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:57.567366    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:57.568050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:57.568014    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:57.568842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:57.568813    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:18:59.993703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:59.993667    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"]
Nov 02 21:19:00.000018 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:18:59.999989    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.044234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.044199    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.044401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.044265    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.044401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.044343    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.044401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.044363    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.044401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.044389    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.044539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.044460    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.044539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.044495    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-kube-api-access-2s2bv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.044539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.044521    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.044647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.044538    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.145882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.145846    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.145882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.145884    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.145913    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.145937    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.145959    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-kube-api-access-2s2bv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.145989    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.146015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.146084    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146492 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.146191    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146492 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.146433    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146492 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.146452    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.146510    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.146543    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.146781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.146761    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.148386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.148361    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.148556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.148536    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.190339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.190304    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"]
Nov 02 21:19:00.199276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.199251    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.199420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.199361    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-kube-api-access-2s2bv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gw8qb\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.312823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.312793    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:00.724187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.724163    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"]
Nov 02 21:19:00.727008 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:19:00.726975    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e12023_6b7c_44a9_b1d1_016fd7e1c43b.slice/crio-3b80bca1be3629fe372316a712b90a94dcd3ab98b32e7c3db0b70a6b053137f0 WatchSource:0}: Error finding container 3b80bca1be3629fe372316a712b90a94dcd3ab98b32e7c3db0b70a6b053137f0: Status 404 returned error can't find the container with id 3b80bca1be3629fe372316a712b90a94dcd3ab98b32e7c3db0b70a6b053137f0
Nov 02 21:19:00.729960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.729927    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:19:00.730041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.729991    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:19:00.730041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:00.730018    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:19:01.319575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:01.319538    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" event={"ID":"22e12023-6b7c-44a9-b1d1-016fd7e1c43b","Type":"ContainerStarted","Data":"42e28435607b4942a4aec95ce65eccc6886d10510f3bb4f91a8628bd9dad4ac6"}
Nov 02 21:19:01.321454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:01.321410    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" event={"ID":"22e12023-6b7c-44a9-b1d1-016fd7e1c43b","Type":"ContainerStarted","Data":"3b80bca1be3629fe372316a712b90a94dcd3ab98b32e7c3db0b70a6b053137f0"}
Nov 02 21:19:01.323039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:01.322980    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:01.324949 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:01.324926    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gw8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 21:19:01.325172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:01.325149    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 21:19:01.527798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:01.527723    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podStartSLOduration=2.5277017710000003 podStartE2EDuration="2.527701771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:01.483326013 +0000 UTC m=+628.746862341" watchObservedRunningTime="2026-11-02 21:19:01.527701771 +0000 UTC m=+628.791238089"
Nov 02 21:19:02.318555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:02.318514    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gw8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 21:19:02.318859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:02.318829    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 21:19:03.320133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:03.319510    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gw8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 21:19:03.320133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:03.319570    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 21:19:04.321133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:04.320526    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gw8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 21:19:04.321133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:04.320602    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 21:19:05.315573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:05.315533    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gw8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 21:19:05.315884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:05.315857    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 21:19:06.319132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:06.318557    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gw8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 21:19:06.319132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:06.318658    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 21:19:07.313279 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:07.313247    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gw8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 21:19:07.313456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:07.313301    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 21:19:08.810792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:08.810758    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:08.811270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:08.811126    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:08.811917 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:08.811897    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:19:12.094859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.094825    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-867cc8b678-sgzth"]
Nov 02 21:19:12.102889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.102867    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-867cc8b678-sgzth"
Nov 02 21:19:12.150584 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.150555    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-867cc8b678-sgzth"]
Nov 02 21:19:12.194492 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.194465    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/774878d1-3662-465a-8a44-3fa542eefd66-tls-cert\") pod \"authorino-867cc8b678-sgzth\" (UID: \"774878d1-3662-465a-8a44-3fa542eefd66\") " pod="kuadrant-system/authorino-867cc8b678-sgzth"
Nov 02 21:19:12.194655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.194503    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hs7x\" (UniqueName: \"kubernetes.io/projected/774878d1-3662-465a-8a44-3fa542eefd66-kube-api-access-2hs7x\") pod \"authorino-867cc8b678-sgzth\" (UID: \"774878d1-3662-465a-8a44-3fa542eefd66\") " pod="kuadrant-system/authorino-867cc8b678-sgzth"
Nov 02 21:19:12.197560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.197541    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tr6zv\""
Nov 02 21:19:12.197616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.197541    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 21:19:12.295230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.295201    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hs7x\" (UniqueName: \"kubernetes.io/projected/774878d1-3662-465a-8a44-3fa542eefd66-kube-api-access-2hs7x\") pod \"authorino-867cc8b678-sgzth\" (UID: \"774878d1-3662-465a-8a44-3fa542eefd66\") " pod="kuadrant-system/authorino-867cc8b678-sgzth"
Nov 02 21:19:12.295400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.295305    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/774878d1-3662-465a-8a44-3fa542eefd66-tls-cert\") pod \"authorino-867cc8b678-sgzth\" (UID: \"774878d1-3662-465a-8a44-3fa542eefd66\") " pod="kuadrant-system/authorino-867cc8b678-sgzth"
Nov 02 21:19:12.297701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.297672    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/774878d1-3662-465a-8a44-3fa542eefd66-tls-cert\") pod \"authorino-867cc8b678-sgzth\" (UID: \"774878d1-3662-465a-8a44-3fa542eefd66\") " pod="kuadrant-system/authorino-867cc8b678-sgzth"
Nov 02 21:19:12.348313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.348246    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hs7x\" (UniqueName: \"kubernetes.io/projected/774878d1-3662-465a-8a44-3fa542eefd66-kube-api-access-2hs7x\") pod \"authorino-867cc8b678-sgzth\" (UID: \"774878d1-3662-465a-8a44-3fa542eefd66\") " pod="kuadrant-system/authorino-867cc8b678-sgzth"
Nov 02 21:19:12.429127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.429079    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-867cc8b678-sgzth"
Nov 02 21:19:12.790741 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.790712    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-867cc8b678-sgzth"]
Nov 02 21:19:12.794243 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:19:12.794216    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod774878d1_3662_465a_8a44_3fa542eefd66.slice/crio-e1601ceb725fe246e65809ecef963b577cde5ff8810b69a5fa65045c40fecfdf WatchSource:0}: Error finding container e1601ceb725fe246e65809ecef963b577cde5ff8810b69a5fa65045c40fecfdf: Status 404 returned error can't find the container with id e1601ceb725fe246e65809ecef963b577cde5ff8810b69a5fa65045c40fecfdf
Nov 02 21:19:12.795760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:12.795743    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:19:13.382234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:13.382198    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-867cc8b678-sgzth" event={"ID":"774878d1-3662-465a-8a44-3fa542eefd66","Type":"ContainerStarted","Data":"e1601ceb725fe246e65809ecef963b577cde5ff8810b69a5fa65045c40fecfdf"}
Nov 02 21:19:14.112619 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:19:14.112582    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:19:14.387793 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:14.387701    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-867cc8b678-sgzth" event={"ID":"774878d1-3662-465a-8a44-3fa542eefd66","Type":"ContainerStarted","Data":"4f233148106e5d210bdf32f722b9200bb67563a55e7ae0bb3e0931e895fcba37"}
Nov 02 21:19:14.495814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:14.495757    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-867cc8b678-sgzth" podStartSLOduration=1.04062312 podStartE2EDuration="2.495744129s" totalImagesPullingTime="1.455121009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:14.494314254 +0000 UTC m=+641.757850571" watchObservedRunningTime="2026-11-02 21:19:14.495744129 +0000 UTC m=+641.759280486"
Nov 02 21:19:44.113088 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:19:44.113051    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:19:44.479155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.479021    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"]
Nov 02 21:19:44.488557 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.488523    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.537952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.537919    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"]
Nov 02 21:19:44.570807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.570779    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5478da31-gw-openshift-default-dockercfg-64mhc\""
Nov 02 21:19:44.585017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.584980    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-podinfo\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.585182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.585022    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-credential-socket\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.585182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.585053    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-envoy\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.585182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.585101    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbdr\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-kube-api-access-zxbdr\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.585301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.585220    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-socket\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.585301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.585243    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-certs\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.585301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.585261    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-data\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.585301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.585294    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istiod-ca-cert\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.585432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.585407    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-token\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.686649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.686614    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-podinfo\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.686858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.686670    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-credential-socket\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.686858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.686706    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-envoy\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.686858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.686769    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbdr\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-kube-api-access-zxbdr\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.686858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.686830    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-socket\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.687103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.686861    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-certs\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.687103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.686887    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-data\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.687103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.686928    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istiod-ca-cert\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.687103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.686970    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-token\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.687337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.687234    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-credential-socket\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.687337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.687277    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-certs\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.687480 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.687458    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-socket\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.688063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.688042    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-data\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.688498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.688475    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istiod-ca-cert\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.689285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.689264    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-envoy\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.689481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.689463    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-podinfo\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.740549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.740488    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-token\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.740674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.740597    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbdr\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-kube-api-access-zxbdr\") pod \"e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") " pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:44.800757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:44.800721    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:45.214975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:45.214582    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"]
Nov 02 21:19:45.219161 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:19:45.219102    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d007aa_3135_4dbb_b99a_cfff7a7b64c5.slice/crio-5353a67cbe43c3cb16adbb5f0f7dc9d60a8dcb9343318634b6351d44236c40dc WatchSource:0}: Error finding container 5353a67cbe43c3cb16adbb5f0f7dc9d60a8dcb9343318634b6351d44236c40dc: Status 404 returned error can't find the container with id 5353a67cbe43c3cb16adbb5f0f7dc9d60a8dcb9343318634b6351d44236c40dc
Nov 02 21:19:45.221813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:45.221772    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:19:45.221909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:45.221846    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:19:45.221909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:45.221881    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:19:45.519982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:45.519945    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" event={"ID":"76d007aa-3135-4dbb-b99a-cfff7a7b64c5","Type":"ContainerStarted","Data":"3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942"}
Nov 02 21:19:45.520203 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:45.519997    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" event={"ID":"76d007aa-3135-4dbb-b99a-cfff7a7b64c5","Type":"ContainerStarted","Data":"5353a67cbe43c3cb16adbb5f0f7dc9d60a8dcb9343318634b6351d44236c40dc"}
Nov 02 21:19:45.631449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:45.631385    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" podStartSLOduration=1.631365831 podStartE2EDuration="1.631365831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:45.62953826 +0000 UTC m=+672.893074602" watchObservedRunningTime="2026-11-02 21:19:45.631365831 +0000 UTC m=+672.894902149"
Nov 02 21:19:45.801245 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:45.801162    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:46.806250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:46.806221    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:47.529129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:47.529081    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:47.530032 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:47.530010    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:19:50.788498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.788469    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"]
Nov 02 21:19:50.792074 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.792048    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.857689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.855176    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-54a47302-openshift-default-dockercfg-5kbdp\""
Nov 02 21:19:50.864090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.864056    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75499c01-f0db-48cf-b2de-33e2fff4502b-istiod-ca-cert\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.864090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.864101    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-socket\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.864363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.864156    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-credential-socket\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.864433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.864419    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-podinfo\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.864506 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.864488    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-data\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.864587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.864529    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-token\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.864587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.864559    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6v8\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-kube-api-access-bq6v8\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.864715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.864598    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-certs\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.864715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.864628    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-envoy\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.903717 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.903678    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"]
Nov 02 21:19:50.965416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965387    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6v8\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-kube-api-access-bq6v8\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.965617 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965426    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-certs\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.965617 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965446    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-envoy\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.965617 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965571    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75499c01-f0db-48cf-b2de-33e2fff4502b-istiod-ca-cert\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.965617 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965613    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-socket\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.965848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965661    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-credential-socket\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.965848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965768    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-podinfo\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.965848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965840    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-data\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.965993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965865    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-certs\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.965993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965881    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-token\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.966101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.965998    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-credential-socket\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.966101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.966062    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-socket\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.966292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.966275    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75499c01-f0db-48cf-b2de-33e2fff4502b-istiod-ca-cert\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.966435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.966416    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-data\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.967753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.967730    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-envoy\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:50.968024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:50.968010    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-podinfo\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:51.020236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:51.020202    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-token\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:51.020847 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:51.020825    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6v8\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-kube-api-access-bq6v8\") pod \"e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") " pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:51.105592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:51.105505    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:51.528327 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:51.528304    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"]
Nov 02 21:19:51.530449 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:19:51.530411    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75499c01_f0db_48cf_b2de_33e2fff4502b.slice/crio-3d68e0970ac04308f775c1017c9c40211b1bf134ce156d67a44c65ecfcd25570 WatchSource:0}: Error finding container 3d68e0970ac04308f775c1017c9c40211b1bf134ce156d67a44c65ecfcd25570: Status 404 returned error can't find the container with id 3d68e0970ac04308f775c1017c9c40211b1bf134ce156d67a44c65ecfcd25570
Nov 02 21:19:51.532690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:51.532658    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:19:51.532779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:51.532723    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:19:51.532779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:51.532752    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:19:51.545963 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:51.545940    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" event={"ID":"75499c01-f0db-48cf-b2de-33e2fff4502b","Type":"ContainerStarted","Data":"3d68e0970ac04308f775c1017c9c40211b1bf134ce156d67a44c65ecfcd25570"}
Nov 02 21:19:52.362425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.362391    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"]
Nov 02 21:19:52.366511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.366489    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:19:52.437131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.430441    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"]
Nov 02 21:19:52.437131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.430699    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5478da31\""
Nov 02 21:19:52.437131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.430810    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5478da31-dockercfg-zwmth\""
Nov 02 21:19:52.482621 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.482591    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5p96\" (UniqueName: \"kubernetes.io/projected/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-kube-api-access-p5p96\") pod \"payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf\" (UID: \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:19:52.482799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.482712    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf\" (UID: \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:19:52.552470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.552434    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" event={"ID":"75499c01-f0db-48cf-b2de-33e2fff4502b","Type":"ContainerStarted","Data":"8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357"}
Nov 02 21:19:52.584129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.584084    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf\" (UID: \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:19:52.584294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.584173    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5p96\" (UniqueName: \"kubernetes.io/projected/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-kube-api-access-p5p96\") pod \"payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf\" (UID: \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:19:52.585052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.585028    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf\" (UID: \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:19:52.640168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.640085    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5p96\" (UniqueName: \"kubernetes.io/projected/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-kube-api-access-p5p96\") pod \"payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf\" (UID: \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:19:52.671652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.671583    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" podStartSLOduration=2.671563282 podStartE2EDuration="2.671563282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:52.665647578 +0000 UTC m=+679.929183896" watchObservedRunningTime="2026-11-02 21:19:52.671563282 +0000 UTC m=+679.935099601"
Nov 02 21:19:52.680240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:52.680213    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:19:53.046342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:53.046314    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"]
Nov 02 21:19:53.049262 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:19:53.049226    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3c720c_c6d6_41aa_9fc8_7cbc388fdca6.slice/crio-1e9a13880b817b5ad2fe7b77d4eed86b2b864d57a852568429a9ee1986033043 WatchSource:0}: Error finding container 1e9a13880b817b5ad2fe7b77d4eed86b2b864d57a852568429a9ee1986033043: Status 404 returned error can't find the container with id 1e9a13880b817b5ad2fe7b77d4eed86b2b864d57a852568429a9ee1986033043
Nov 02 21:19:53.106031 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:53.106000    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:53.111255 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:53.111233    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:53.559170 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:53.559131    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf" event={"ID":"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6","Type":"ContainerStarted","Data":"25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0"}
Nov 02 21:19:53.559669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:53.559189    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf" event={"ID":"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6","Type":"ContainerStarted","Data":"1e9a13880b817b5ad2fe7b77d4eed86b2b864d57a852568429a9ee1986033043"}
Nov 02 21:19:53.559669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:53.559434    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:53.560318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:53.560295    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:19:53.666573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:53.666524    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf" podStartSLOduration=1.6665109679999999 podStartE2EDuration="1.666510968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:53.665062967 +0000 UTC m=+680.928599286" watchObservedRunningTime="2026-11-02 21:19:53.666510968 +0000 UTC m=+680.930047284"
Nov 02 21:19:54.564602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:54.564575    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:19:56.830668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:56.830635    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"]
Nov 02 21:19:56.834339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:56.834318    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:19:56.884545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:56.884495    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6f4fe932\""
Nov 02 21:19:56.933502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:56.931378    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6f4fe932-dockercfg-q9nhw\""
Nov 02 21:19:56.936694 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:56.936653    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f366747-a317-4e69-b600-37643e5cc23a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5\" (UID: \"4f366747-a317-4e69-b600-37643e5cc23a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:19:56.936874 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:56.936852    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/4f366747-a317-4e69-b600-37643e5cc23a-kube-api-access-k6chk\") pod \"payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5\" (UID: \"4f366747-a317-4e69-b600-37643e5cc23a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:19:56.942615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:56.942588    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"]
Nov 02 21:19:56.993050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:56.993018    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"]
Nov 02 21:19:56.996874 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:56.996848    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:19:57.038228 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.038197    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/4f366747-a317-4e69-b600-37643e5cc23a-kube-api-access-k6chk\") pod \"payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5\" (UID: \"4f366747-a317-4e69-b600-37643e5cc23a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:19:57.038403 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.038263    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f366747-a317-4e69-b600-37643e5cc23a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5\" (UID: \"4f366747-a317-4e69-b600-37643e5cc23a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:19:57.038994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.038973    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f366747-a317-4e69-b600-37643e5cc23a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5\" (UID: \"4f366747-a317-4e69-b600-37643e5cc23a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:19:57.051236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.051211    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"]
Nov 02 21:19:57.092716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.092647    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/4f366747-a317-4e69-b600-37643e5cc23a-kube-api-access-k6chk\") pod \"payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5\" (UID: \"4f366747-a317-4e69-b600-37643e5cc23a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:19:57.139558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.139523    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpkd\" (UniqueName: \"kubernetes.io/projected/9dcb42db-eb12-4bd6-a0f4-39025f99377d-kube-api-access-jhpkd\") pod \"payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp\" (UID: \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\") " pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:19:57.139692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.139578    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcb42db-eb12-4bd6-a0f4-39025f99377d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp\" (UID: \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\") " pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:19:57.147521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.147501    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:19:57.241134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.241064    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpkd\" (UniqueName: \"kubernetes.io/projected/9dcb42db-eb12-4bd6-a0f4-39025f99377d-kube-api-access-jhpkd\") pod \"payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp\" (UID: \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\") " pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:19:57.241303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.241227    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcb42db-eb12-4bd6-a0f4-39025f99377d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp\" (UID: \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\") " pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:19:57.241959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.241938    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcb42db-eb12-4bd6-a0f4-39025f99377d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp\" (UID: \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\") " pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:19:57.297882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.297848    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpkd\" (UniqueName: \"kubernetes.io/projected/9dcb42db-eb12-4bd6-a0f4-39025f99377d-kube-api-access-jhpkd\") pod \"payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp\" (UID: \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\") " pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:19:57.310004 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.309976    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:19:57.521532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.521507    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"]
Nov 02 21:19:57.523545 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:19:57.523517    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f366747_a317_4e69_b600_37643e5cc23a.slice/crio-406d2825d041a44a56c77e1bf9037d1d43c413ba91cb7ac677793a059183ecb1 WatchSource:0}: Error finding container 406d2825d041a44a56c77e1bf9037d1d43c413ba91cb7ac677793a059183ecb1: Status 404 returned error can't find the container with id 406d2825d041a44a56c77e1bf9037d1d43c413ba91cb7ac677793a059183ecb1
Nov 02 21:19:57.579903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.579877    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5" event={"ID":"4f366747-a317-4e69-b600-37643e5cc23a","Type":"ContainerStarted","Data":"406d2825d041a44a56c77e1bf9037d1d43c413ba91cb7ac677793a059183ecb1"}
Nov 02 21:19:57.685152 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:57.685047    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"]
Nov 02 21:19:57.688497 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:19:57.688470    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcb42db_eb12_4bd6_a0f4_39025f99377d.slice/crio-8f16a9098e1e024caa3736232bfe2b172c48dc46f35c945e74b48641b35e4896 WatchSource:0}: Error finding container 8f16a9098e1e024caa3736232bfe2b172c48dc46f35c945e74b48641b35e4896: Status 404 returned error can't find the container with id 8f16a9098e1e024caa3736232bfe2b172c48dc46f35c945e74b48641b35e4896
Nov 02 21:19:58.587265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:58.587226    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5" event={"ID":"4f366747-a317-4e69-b600-37643e5cc23a","Type":"ContainerStarted","Data":"5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863"}
Nov 02 21:19:58.587825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:58.587399    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:19:58.588729 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:58.588706    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp" event={"ID":"9dcb42db-eb12-4bd6-a0f4-39025f99377d","Type":"ContainerStarted","Data":"16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6"}
Nov 02 21:19:58.588824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:58.588743    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp" event={"ID":"9dcb42db-eb12-4bd6-a0f4-39025f99377d","Type":"ContainerStarted","Data":"8f16a9098e1e024caa3736232bfe2b172c48dc46f35c945e74b48641b35e4896"}
Nov 02 21:19:58.588861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:58.588830    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:19:58.804618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:58.804560    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp" podStartSLOduration=2.804545158 podStartE2EDuration="2.804545158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:58.803126509 +0000 UTC m=+686.066662818" watchObservedRunningTime="2026-11-02 21:19:58.804545158 +0000 UTC m=+686.068081474"
Nov 02 21:19:58.805412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:58.805373    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5" podStartSLOduration=2.805360911 podStartE2EDuration="2.805360911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:58.697120956 +0000 UTC m=+685.960657269" watchObservedRunningTime="2026-11-02 21:19:58.805360911 +0000 UTC m=+686.068897255"
Nov 02 21:19:59.206199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.206150    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"]
Nov 02 21:19:59.211053 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.211030    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:19:59.258850 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.258824    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p546v\" (UniqueName: \"kubernetes.io/projected/b8f23572-02ac-4561-aa30-4c6ba90cd487-kube-api-access-p546v\") pod \"payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq\" (UID: \"b8f23572-02ac-4561-aa30-4c6ba90cd487\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:19:59.259005 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.258866    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f23572-02ac-4561-aa30-4c6ba90cd487-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq\" (UID: \"b8f23572-02ac-4561-aa30-4c6ba90cd487\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:19:59.262615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.262591    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-170b4fea\""
Nov 02 21:19:59.264797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.264779    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-170b4fea-dockercfg-rhkt4\""
Nov 02 21:19:59.310955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.310922    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"]
Nov 02 21:19:59.341564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.341535    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"]
Nov 02 21:19:59.345202 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.345180    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:19:59.359329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.359303    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs26l\" (UniqueName: \"kubernetes.io/projected/9903f562-246b-4552-9d87-b03f1ace9b18-kube-api-access-rs26l\") pod \"payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk\" (UID: \"9903f562-246b-4552-9d87-b03f1ace9b18\") " pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:19:59.359445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.359335    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903f562-246b-4552-9d87-b03f1ace9b18-plugins-config-volume\") pod \"payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk\" (UID: \"9903f562-246b-4552-9d87-b03f1ace9b18\") " pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:19:59.359515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.359495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p546v\" (UniqueName: \"kubernetes.io/projected/b8f23572-02ac-4561-aa30-4c6ba90cd487-kube-api-access-p546v\") pod \"payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq\" (UID: \"b8f23572-02ac-4561-aa30-4c6ba90cd487\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:19:59.359570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.359556    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f23572-02ac-4561-aa30-4c6ba90cd487-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq\" (UID: \"b8f23572-02ac-4561-aa30-4c6ba90cd487\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:19:59.360217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.360199    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f23572-02ac-4561-aa30-4c6ba90cd487-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq\" (UID: \"b8f23572-02ac-4561-aa30-4c6ba90cd487\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:19:59.399816 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.399788    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"]
Nov 02 21:19:59.414551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.414524    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p546v\" (UniqueName: \"kubernetes.io/projected/b8f23572-02ac-4561-aa30-4c6ba90cd487-kube-api-access-p546v\") pod \"payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq\" (UID: \"b8f23572-02ac-4561-aa30-4c6ba90cd487\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:19:59.460331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.460234    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs26l\" (UniqueName: \"kubernetes.io/projected/9903f562-246b-4552-9d87-b03f1ace9b18-kube-api-access-rs26l\") pod \"payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk\" (UID: \"9903f562-246b-4552-9d87-b03f1ace9b18\") " pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:19:59.460331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.460283    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903f562-246b-4552-9d87-b03f1ace9b18-plugins-config-volume\") pod \"payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk\" (UID: \"9903f562-246b-4552-9d87-b03f1ace9b18\") " pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:19:59.469063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.461211    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903f562-246b-4552-9d87-b03f1ace9b18-plugins-config-volume\") pod \"payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk\" (UID: \"9903f562-246b-4552-9d87-b03f1ace9b18\") " pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:19:59.519084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.519047    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs26l\" (UniqueName: \"kubernetes.io/projected/9903f562-246b-4552-9d87-b03f1ace9b18-kube-api-access-rs26l\") pod \"payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk\" (UID: \"9903f562-246b-4552-9d87-b03f1ace9b18\") " pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:19:59.523590 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.523567    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:19:59.661820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.661784    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:19:59.888180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:19:59.887859    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"]
Nov 02 21:19:59.892035 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:19:59.892004    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f23572_02ac_4561_aa30_4c6ba90cd487.slice/crio-20783b270d2e99a6e9d73f4cf0a5de4aca5cbbbc0156490cb827f2fd35f110f2 WatchSource:0}: Error finding container 20783b270d2e99a6e9d73f4cf0a5de4aca5cbbbc0156490cb827f2fd35f110f2: Status 404 returned error can't find the container with id 20783b270d2e99a6e9d73f4cf0a5de4aca5cbbbc0156490cb827f2fd35f110f2
Nov 02 21:20:00.027139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:00.027097    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"]
Nov 02 21:20:00.029632 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:00.029595    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9903f562_246b_4552_9d87_b03f1ace9b18.slice/crio-09d9ee44a23021a5b86220124b1ca18e51e7b4a70be2182f80ca8f4138d29d24 WatchSource:0}: Error finding container 09d9ee44a23021a5b86220124b1ca18e51e7b4a70be2182f80ca8f4138d29d24: Status 404 returned error can't find the container with id 09d9ee44a23021a5b86220124b1ca18e51e7b4a70be2182f80ca8f4138d29d24
Nov 02 21:20:00.599516 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:00.599477    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq" event={"ID":"b8f23572-02ac-4561-aa30-4c6ba90cd487","Type":"ContainerStarted","Data":"f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced"}
Nov 02 21:20:00.599718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:00.599535    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq" event={"ID":"b8f23572-02ac-4561-aa30-4c6ba90cd487","Type":"ContainerStarted","Data":"20783b270d2e99a6e9d73f4cf0a5de4aca5cbbbc0156490cb827f2fd35f110f2"}
Nov 02 21:20:00.599718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:00.599583    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:20:00.601189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:00.601160    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk" event={"ID":"9903f562-246b-4552-9d87-b03f1ace9b18","Type":"ContainerStarted","Data":"659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb"}
Nov 02 21:20:00.601313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:00.601207    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk" event={"ID":"9903f562-246b-4552-9d87-b03f1ace9b18","Type":"ContainerStarted","Data":"09d9ee44a23021a5b86220124b1ca18e51e7b4a70be2182f80ca8f4138d29d24"}
Nov 02 21:20:00.601313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:00.601254    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:20:00.709369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:00.709314    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq" podStartSLOduration=1.709300974 podStartE2EDuration="1.709300974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:00.708395071 +0000 UTC m=+687.971931389" watchObservedRunningTime="2026-11-02 21:20:00.709300974 +0000 UTC m=+687.972837292"
Nov 02 21:20:00.815194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:00.815142    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk" podStartSLOduration=1.815101271 podStartE2EDuration="1.815101271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:00.81497781 +0000 UTC m=+688.078514153" watchObservedRunningTime="2026-11-02 21:20:00.815101271 +0000 UTC m=+688.078637587"
Nov 02 21:20:01.633944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.633911    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"]
Nov 02 21:20:01.637126 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.637095    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:01.679636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.679599    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmw2c\" (UniqueName: \"kubernetes.io/projected/d9762ede-8d11-4e06-a48e-f2f82e472d36-kube-api-access-pmw2c\") pod \"payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb\" (UID: \"d9762ede-8d11-4e06-a48e-f2f82e472d36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:01.679815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.679792    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9762ede-8d11-4e06-a48e-f2f82e472d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb\" (UID: \"d9762ede-8d11-4e06-a48e-f2f82e472d36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:01.688029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.688006    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-84de7f3a\""
Nov 02 21:20:01.734072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.734038    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-84de7f3a-dockercfg-pvffw\""
Nov 02 21:20:01.736862 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.736832    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"]
Nov 02 21:20:01.780569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.780536    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmw2c\" (UniqueName: \"kubernetes.io/projected/d9762ede-8d11-4e06-a48e-f2f82e472d36-kube-api-access-pmw2c\") pod \"payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb\" (UID: \"d9762ede-8d11-4e06-a48e-f2f82e472d36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:01.780737 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.780626    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9762ede-8d11-4e06-a48e-f2f82e472d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb\" (UID: \"d9762ede-8d11-4e06-a48e-f2f82e472d36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:01.781399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.781381    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9762ede-8d11-4e06-a48e-f2f82e472d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb\" (UID: \"d9762ede-8d11-4e06-a48e-f2f82e472d36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:01.814011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.813974    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"]
Nov 02 21:20:01.817764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.817745    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:01.836521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.836499    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmw2c\" (UniqueName: \"kubernetes.io/projected/d9762ede-8d11-4e06-a48e-f2f82e472d36-kube-api-access-pmw2c\") pod \"payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb\" (UID: \"d9762ede-8d11-4e06-a48e-f2f82e472d36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:01.869664 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.869634    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"]
Nov 02 21:20:01.881227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.881200    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18abb932-9f6d-49e1-ab01-b1d9774e7a00-plugins-config-volume\") pod \"payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw\" (UID: \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\") " pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:01.881362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.881273    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpl5\" (UniqueName: \"kubernetes.io/projected/18abb932-9f6d-49e1-ab01-b1d9774e7a00-kube-api-access-vlpl5\") pod \"payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw\" (UID: \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\") " pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:01.950278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.950206    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:01.982526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.982501    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpl5\" (UniqueName: \"kubernetes.io/projected/18abb932-9f6d-49e1-ab01-b1d9774e7a00-kube-api-access-vlpl5\") pod \"payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw\" (UID: \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\") " pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:01.982700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.982686    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18abb932-9f6d-49e1-ab01-b1d9774e7a00-plugins-config-volume\") pod \"payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw\" (UID: \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\") " pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:01.983368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:01.983349    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18abb932-9f6d-49e1-ab01-b1d9774e7a00-plugins-config-volume\") pod \"payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw\" (UID: \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\") " pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:02.035778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.035752    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpl5\" (UniqueName: \"kubernetes.io/projected/18abb932-9f6d-49e1-ab01-b1d9774e7a00-kube-api-access-vlpl5\") pod \"payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw\" (UID: \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\") " pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:02.130522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.130489    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:02.312710 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.312687    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"]
Nov 02 21:20:02.314737 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:02.314709    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9762ede_8d11_4e06_a48e_f2f82e472d36.slice/crio-ed1e1dd08f75f44726e3988dcdb654fe5eb2889c0d196685dd09b324eb4aed1b WatchSource:0}: Error finding container ed1e1dd08f75f44726e3988dcdb654fe5eb2889c0d196685dd09b324eb4aed1b: Status 404 returned error can't find the container with id ed1e1dd08f75f44726e3988dcdb654fe5eb2889c0d196685dd09b324eb4aed1b
Nov 02 21:20:02.492091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.492068    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"]
Nov 02 21:20:02.494086 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:02.494049    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18abb932_9f6d_49e1_ab01_b1d9774e7a00.slice/crio-0d0375454afd61f27e12d33f647e409e1fff6cdb520aac8e15e093827ab20fbc WatchSource:0}: Error finding container 0d0375454afd61f27e12d33f647e409e1fff6cdb520aac8e15e093827ab20fbc: Status 404 returned error can't find the container with id 0d0375454afd61f27e12d33f647e409e1fff6cdb520aac8e15e093827ab20fbc
Nov 02 21:20:02.611695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.611617    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw" event={"ID":"18abb932-9f6d-49e1-ab01-b1d9774e7a00","Type":"ContainerStarted","Data":"f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff"}
Nov 02 21:20:02.611695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.611692    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw" event={"ID":"18abb932-9f6d-49e1-ab01-b1d9774e7a00","Type":"ContainerStarted","Data":"0d0375454afd61f27e12d33f647e409e1fff6cdb520aac8e15e093827ab20fbc"}
Nov 02 21:20:02.611893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.611718    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:02.613151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.613101    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb" event={"ID":"d9762ede-8d11-4e06-a48e-f2f82e472d36","Type":"ContainerStarted","Data":"0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94"}
Nov 02 21:20:02.613256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.613167    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb" event={"ID":"d9762ede-8d11-4e06-a48e-f2f82e472d36","Type":"ContainerStarted","Data":"ed1e1dd08f75f44726e3988dcdb654fe5eb2889c0d196685dd09b324eb4aed1b"}
Nov 02 21:20:02.613308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.613262    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:02.719938 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.719804    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw" podStartSLOduration=1.7197827 podStartE2EDuration="1.7197827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:02.71830261 +0000 UTC m=+689.981838928" watchObservedRunningTime="2026-11-02 21:20:02.7197827 +0000 UTC m=+689.983319018"
Nov 02 21:20:02.829975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:02.829914    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb" podStartSLOduration=1.829897135 podStartE2EDuration="1.829897135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:02.827983888 +0000 UTC m=+690.091520205" watchObservedRunningTime="2026-11-02 21:20:02.829897135 +0000 UTC m=+690.093433456"
Nov 02 21:20:05.572590 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:05.572562    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:20:07.367843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:07.367805    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:20:09.596907 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:09.596878    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:20:09.597376 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:09.596946    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:20:11.627211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:11.627179    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:20:11.627716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:11.627686    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:20:13.622731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:13.622703    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:20:13.623199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:13.622767    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:20:14.113428 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:20:14.113389    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:20:17.531240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:17.531200    4528 patch_prober.go:29] interesting pod/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 21:20:17.531740 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:17.531269    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 21:20:17.692089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:17.692060    4528 generic.go:356] "Generic (PLEG): container finished" podID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerID="3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942" exitCode=0
Nov 02 21:20:17.692261 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:17.692138    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" event={"ID":"76d007aa-3135-4dbb-b99a-cfff7a7b64c5","Type":"ContainerDied","Data":"3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942"}
Nov 02 21:20:17.692502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:17.692488    4528 scope.go:122] "RemoveContainer" containerID="3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942"
Nov 02 21:20:17.693739 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:17.693715    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:20:17.693809 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:17.693781    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:20:17.693848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:17.693810    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:20:18.698133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:18.698076    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" event={"ID":"76d007aa-3135-4dbb-b99a-cfff7a7b64c5","Type":"ContainerStarted","Data":"05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74"}
Nov 02 21:20:18.811555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:18.811520    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:20:18.811899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:18.811873    4528 patch_prober.go:29] interesting pod/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 21:20:18.812050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:18.812027    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 21:20:19.811134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:19.801289    4528 patch_prober.go:29] interesting pod/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 21:20:19.811134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:19.801348    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 21:20:20.801163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:20.801092    4528 patch_prober.go:29] interesting pod/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 21:20:20.801384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:20.801185    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 21:20:21.825133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:21.824235    4528 patch_prober.go:29] interesting pod/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 21:20:21.825133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:21.824301    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 21:20:22.801268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:22.801233    4528 patch_prober.go:29] interesting pod/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 21:20:22.801566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:22.801539    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 21:20:23.806015 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:23.805986    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:20:23.806524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:23.806402    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:20:23.807092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:23.807074    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:20:27.065474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:27.065438    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:20:34.575966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.575935    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"]
Nov 02 21:20:34.580683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.580661    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.676692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.676660    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:20:34.676692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.676677    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:20:34.676692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.676660    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:20:34.677026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.676740    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:20:34.678806 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.678779    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"]
Nov 02 21:20:34.700430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.700397    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zmkj\" (UniqueName: \"kubernetes.io/projected/4a2ad085-4c35-4895-9c69-a48439905d1e-kube-api-access-9zmkj\") pod \"maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.700582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.700558    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.700759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.700739    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.802036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.802005    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.802280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.802045    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zmkj\" (UniqueName: \"kubernetes.io/projected/4a2ad085-4c35-4895-9c69-a48439905d1e-kube-api-access-9zmkj\") pod \"maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.802280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.802098    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.802894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.802862    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.804318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.804302    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.855729 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.855642    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zmkj\" (UniqueName: \"kubernetes.io/projected/4a2ad085-4c35-4895-9c69-a48439905d1e-kube-api-access-9zmkj\") pod \"maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:34.893552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:34.893513    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:35.262064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:35.262041    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"]
Nov 02 21:20:35.265431 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:35.265400    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2ad085_4c35_4895_9c69_a48439905d1e.slice/crio-bc110de45accbc39438aec7f09bea2c05ba471984da7c1e50279fd2e5d774456 WatchSource:0}: Error finding container bc110de45accbc39438aec7f09bea2c05ba471984da7c1e50279fd2e5d774456: Status 404 returned error can't find the container with id bc110de45accbc39438aec7f09bea2c05ba471984da7c1e50279fd2e5d774456
Nov 02 21:20:35.828489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:35.828450    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj" event={"ID":"4a2ad085-4c35-4895-9c69-a48439905d1e","Type":"ContainerStarted","Data":"bc110de45accbc39438aec7f09bea2c05ba471984da7c1e50279fd2e5d774456"}
Nov 02 21:20:38.738295 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.738263    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"]
Nov 02 21:20:38.745061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.745037    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:38.794543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.794512    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"]
Nov 02 21:20:38.855573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.855541    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:38.855744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.855624    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jlp\" (UniqueName: \"kubernetes.io/projected/bbd75759-e7c3-4950-82a0-3c6c4b174e87-kube-api-access-l6jlp\") pod \"maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:38.855791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.855746    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:38.956604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.956567    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:38.956772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.956636    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jlp\" (UniqueName: \"kubernetes.io/projected/bbd75759-e7c3-4950-82a0-3c6c4b174e87-kube-api-access-l6jlp\") pod \"maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:38.956772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.956671    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:38.957305 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.957264    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:38.959179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:38.959138    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:39.028492 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:39.028464    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jlp\" (UniqueName: \"kubernetes.io/projected/bbd75759-e7c3-4950-82a0-3c6c4b174e87-kube-api-access-l6jlp\") pod \"maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:39.058693 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:39.058659    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:39.452737 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:39.452711    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"]
Nov 02 21:20:39.455138 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:39.455086    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd75759_e7c3_4950_82a0_3c6c4b174e87.slice/crio-c8b155bd2130ff790232da8bdc8c5a901145596b71ae862e7f2b51cd2529db0f WatchSource:0}: Error finding container c8b155bd2130ff790232da8bdc8c5a901145596b71ae862e7f2b51cd2529db0f: Status 404 returned error can't find the container with id c8b155bd2130ff790232da8bdc8c5a901145596b71ae862e7f2b51cd2529db0f
Nov 02 21:20:39.859356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:39.859323    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk" event={"ID":"bbd75759-e7c3-4950-82a0-3c6c4b174e87","Type":"ContainerStarted","Data":"c8b155bd2130ff790232da8bdc8c5a901145596b71ae862e7f2b51cd2529db0f"}
Nov 02 21:20:40.865805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:40.865719    4528 generic.go:356] "Generic (PLEG): container finished" podID="4a2ad085-4c35-4895-9c69-a48439905d1e" containerID="7d5e377bdae0893233d71a3e090ea76cde7ab6d75a69f103693c564c8d88a3b3" exitCode=0
Nov 02 21:20:40.865805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:40.865776    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj" event={"ID":"4a2ad085-4c35-4895-9c69-a48439905d1e","Type":"ContainerDied","Data":"7d5e377bdae0893233d71a3e090ea76cde7ab6d75a69f103693c564c8d88a3b3"}
Nov 02 21:20:40.964692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:40.964646    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:20:41.872219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:41.872180    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk" event={"ID":"bbd75759-e7c3-4950-82a0-3c6c4b174e87","Type":"ContainerStarted","Data":"7811e40bacac6f03b1bec63e1e4ea9dc99cee2bc92f8950aca367936ceb1bcf6"}
Nov 02 21:20:42.026023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.025993    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:42.104903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.104869    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2ad085-4c35-4895-9c69-a48439905d1e-kube-api-access-9zmkj\" (UniqueName: \"kubernetes.io/projected/4a2ad085-4c35-4895-9c69-a48439905d1e-kube-api-access-9zmkj\") pod \"4a2ad085-4c35-4895-9c69-a48439905d1e\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") "
Nov 02 21:20:42.105089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.104969    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-openshift-ca\") pod \"4a2ad085-4c35-4895-9c69-a48439905d1e\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") "
Nov 02 21:20:42.105089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.105039    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-xks-ca\") pod \"4a2ad085-4c35-4895-9c69-a48439905d1e\" (UID: \"4a2ad085-4c35-4895-9c69-a48439905d1e\") "
Nov 02 21:20:42.105507 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.105479    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-openshift-ca" pod "4a2ad085-4c35-4895-9c69-a48439905d1e" (UID: "4a2ad085-4c35-4895-9c69-a48439905d1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:20:42.107413 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.107384    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-xks-ca" pod "4a2ad085-4c35-4895-9c69-a48439905d1e" (UID: "4a2ad085-4c35-4895-9c69-a48439905d1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:20:42.107497 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.107393    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2ad085-4c35-4895-9c69-a48439905d1e-kube-api-access-9zmkj" pod "4a2ad085-4c35-4895-9c69-a48439905d1e" (UID: "4a2ad085-4c35-4895-9c69-a48439905d1e"). InnerVolumeSpecName "kube-api-access-9zmkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:20:42.135509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.134402    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk" podStartSLOduration=2.314149823 podStartE2EDuration="4.134382537s" totalImagesPullingTime="1.820232714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:41.997851928 +0000 UTC m=+729.261388246" watchObservedRunningTime="2026-11-02 21:20:42.134382537 +0000 UTC m=+729.397918855"
Nov 02 21:20:42.206770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.206734    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zmkj\" (UniqueName: \"kubernetes.io/projected/4a2ad085-4c35-4895-9c69-a48439905d1e-kube-api-access-9zmkj\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:42.206770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.206770    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:42.206969 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.206786    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2ad085-4c35-4895-9c69-a48439905d1e-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:42.878483 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.878450    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"
Nov 02 21:20:42.878944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.878449    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj" event={"ID":"4a2ad085-4c35-4895-9c69-a48439905d1e","Type":"ContainerDied","Data":"bc110de45accbc39438aec7f09bea2c05ba471984da7c1e50279fd2e5d774456"}
Nov 02 21:20:42.878944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.878784    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc110de45accbc39438aec7f09bea2c05ba471984da7c1e50279fd2e5d774456"
Nov 02 21:20:42.880448 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.880417    4528 generic.go:356] "Generic (PLEG): container finished" podID="bbd75759-e7c3-4950-82a0-3c6c4b174e87" containerID="7811e40bacac6f03b1bec63e1e4ea9dc99cee2bc92f8950aca367936ceb1bcf6" exitCode=0
Nov 02 21:20:42.880570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:42.880498    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk" event={"ID":"bbd75759-e7c3-4950-82a0-3c6c4b174e87","Type":"ContainerDied","Data":"7811e40bacac6f03b1bec63e1e4ea9dc99cee2bc92f8950aca367936ceb1bcf6"}
Nov 02 21:20:44.034763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.034737    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:44.111698 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:20:44.111661    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:20:44.125148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.125077    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-xks-ca\") pod \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") "
Nov 02 21:20:44.125253 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.125171    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-openshift-ca\") pod \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") "
Nov 02 21:20:44.125308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.125292    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd75759-e7c3-4950-82a0-3c6c4b174e87-kube-api-access-l6jlp\" (UniqueName: \"kubernetes.io/projected/bbd75759-e7c3-4950-82a0-3c6c4b174e87-kube-api-access-l6jlp\") pod \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\" (UID: \"bbd75759-e7c3-4950-82a0-3c6c4b174e87\") "
Nov 02 21:20:44.125518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.125491    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-openshift-ca" pod "bbd75759-e7c3-4950-82a0-3c6c4b174e87" (UID: "bbd75759-e7c3-4950-82a0-3c6c4b174e87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:20:44.125633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.125619    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:44.127173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.127149    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd75759-e7c3-4950-82a0-3c6c4b174e87-kube-api-access-l6jlp" pod "bbd75759-e7c3-4950-82a0-3c6c4b174e87" (UID: "bbd75759-e7c3-4950-82a0-3c6c4b174e87"). InnerVolumeSpecName "kube-api-access-l6jlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:20:44.127173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.127156    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-xks-ca" pod "bbd75759-e7c3-4950-82a0-3c6c4b174e87" (UID: "bbd75759-e7c3-4950-82a0-3c6c4b174e87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:20:44.227180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.227068    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd75759-e7c3-4950-82a0-3c6c4b174e87-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:44.227180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.227094    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6jlp\" (UniqueName: \"kubernetes.io/projected/bbd75759-e7c3-4950-82a0-3c6c4b174e87-kube-api-access-l6jlp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:44.891027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.890996    4528 generic.go:356] "Generic (PLEG): container finished" podID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerID="8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357" exitCode=0
Nov 02 21:20:44.891246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.891074    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" event={"ID":"75499c01-f0db-48cf-b2de-33e2fff4502b","Type":"ContainerDied","Data":"8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357"}
Nov 02 21:20:44.891562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.891534    4528 scope.go:122] "RemoveContainer" containerID="8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357"
Nov 02 21:20:44.892534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.892515    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"
Nov 02 21:20:44.892534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.892527    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk" event={"ID":"bbd75759-e7c3-4950-82a0-3c6c4b174e87","Type":"ContainerDied","Data":"c8b155bd2130ff790232da8bdc8c5a901145596b71ae862e7f2b51cd2529db0f"}
Nov 02 21:20:44.892669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.892566    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b155bd2130ff790232da8bdc8c5a901145596b71ae862e7f2b51cd2529db0f"
Nov 02 21:20:44.892910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.892885    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:20:44.893024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.892998    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:20:44.893088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:44.893067    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:20:45.899443 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:45.899405    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" event={"ID":"75499c01-f0db-48cf-b2de-33e2fff4502b","Type":"ContainerStarted","Data":"457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36"}
Nov 02 21:20:46.114134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.112136    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:20:46.114134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.112269    4528 patch_prober.go:29] interesting pod/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 21:20:46.114134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.112314    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 21:20:46.441491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.441456    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"]
Nov 02 21:20:46.539299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.539265    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"]
Nov 02 21:20:46.539761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.539743    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bbd75759-e7c3-4950-82a0-3c6c4b174e87" containerName="revoke-keys"
Nov 02 21:20:46.539855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.539762    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd75759-e7c3-4950-82a0-3c6c4b174e87" containerName="revoke-keys"
Nov 02 21:20:46.539855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.539792    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4a2ad085-4c35-4895-9c69-a48439905d1e" containerName="revoke-keys"
Nov 02 21:20:46.539855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.539800    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2ad085-4c35-4895-9c69-a48439905d1e" containerName="revoke-keys"
Nov 02 21:20:46.540047 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.539911    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd75759-e7c3-4950-82a0-3c6c4b174e87" containerName="revoke-keys"
Nov 02 21:20:46.540047 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.539926    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2ad085-4c35-4895-9c69-a48439905d1e" containerName="revoke-keys"
Nov 02 21:20:46.543412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.543389    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.545322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.545295    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"]
Nov 02 21:20:46.594019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.593986    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:20:46.594019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.593996    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:20:46.594268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.594041    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:20:46.594268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.594043    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:20:46.595864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.595841    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"]
Nov 02 21:20:46.650949 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.650912    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.651149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.651058    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzfjb\" (UniqueName: \"kubernetes.io/projected/842b3992-d92c-4460-8f3c-c6e5c42e03d1-kube-api-access-rzfjb\") pod \"maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.651243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.651219    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.752538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.752455    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.752697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.752546    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzfjb\" (UniqueName: \"kubernetes.io/projected/842b3992-d92c-4460-8f3c-c6e5c42e03d1-kube-api-access-rzfjb\") pod \"maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.752697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.752597    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.753129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.753083    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.754839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.754817    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.828700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.828670    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzfjb\" (UniqueName: \"kubernetes.io/projected/842b3992-d92c-4460-8f3c-c6e5c42e03d1-kube-api-access-rzfjb\") pod \"maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:46.859284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:46.859250    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:47.125133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:47.118544    4528 patch_prober.go:29] interesting pod/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 21:20:47.125133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:47.118625    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 21:20:47.249779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:47.249752    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"]
Nov 02 21:20:47.252761 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:47.252734    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842b3992_d92c_4460_8f3c_c6e5c42e03d1.slice/crio-aaee6abb26223e16ea5c5ca7af974ee7d9c9e41ed54afaa8f4c5a6344e1fb2a6 WatchSource:0}: Error finding container aaee6abb26223e16ea5c5ca7af974ee7d9c9e41ed54afaa8f4c5a6344e1fb2a6: Status 404 returned error can't find the container with id aaee6abb26223e16ea5c5ca7af974ee7d9c9e41ed54afaa8f4c5a6344e1fb2a6
Nov 02 21:20:47.938727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:47.938687    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b" event={"ID":"842b3992-d92c-4460-8f3c-c6e5c42e03d1","Type":"ContainerStarted","Data":"b520cb7b0c5e60635caf7c1c01f08a40d546abc368eb8799cb71579049afad2b"}
Nov 02 21:20:47.938898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:47.938736    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b" event={"ID":"842b3992-d92c-4460-8f3c-c6e5c42e03d1","Type":"ContainerStarted","Data":"aaee6abb26223e16ea5c5ca7af974ee7d9c9e41ed54afaa8f4c5a6344e1fb2a6"}
Nov 02 21:20:48.047775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:48.047723    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b" podStartSLOduration=2.047708647 podStartE2EDuration="2.047708647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:48.046100425 +0000 UTC m=+735.309636743" watchObservedRunningTime="2026-11-02 21:20:48.047708647 +0000 UTC m=+735.311244963"
Nov 02 21:20:48.109134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:48.108524    4528 patch_prober.go:29] interesting pod/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 21:20:48.109134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:48.108584    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 21:20:48.944043 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:48.944006    4528 generic.go:356] "Generic (PLEG): container finished" podID="842b3992-d92c-4460-8f3c-c6e5c42e03d1" containerID="b520cb7b0c5e60635caf7c1c01f08a40d546abc368eb8799cb71579049afad2b" exitCode=0
Nov 02 21:20:48.944445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:48.944054    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b" event={"ID":"842b3992-d92c-4460-8f3c-c6e5c42e03d1","Type":"ContainerDied","Data":"b520cb7b0c5e60635caf7c1c01f08a40d546abc368eb8799cb71579049afad2b"}
Nov 02 21:20:49.122136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:49.110536    4528 patch_prober.go:29] interesting pod/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 21:20:49.122136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:49.110599    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 21:20:50.094273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.094250    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:50.126161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.125540    4528 patch_prober.go:29] interesting pod/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 21:20:50.126161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.125599    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 21:20:50.192025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.191995    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-openshift-ca\") pod \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") "
Nov 02 21:20:50.192234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.192050    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-xks-ca\") pod \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") "
Nov 02 21:20:50.192234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.192099    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842b3992-d92c-4460-8f3c-c6e5c42e03d1-kube-api-access-rzfjb\" (UniqueName: \"kubernetes.io/projected/842b3992-d92c-4460-8f3c-c6e5c42e03d1-kube-api-access-rzfjb\") pod \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\" (UID: \"842b3992-d92c-4460-8f3c-c6e5c42e03d1\") "
Nov 02 21:20:50.192416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.192393    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-openshift-ca" pod "842b3992-d92c-4460-8f3c-c6e5c42e03d1" (UID: "842b3992-d92c-4460-8f3c-c6e5c42e03d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:20:50.194196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.194152    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-xks-ca" pod "842b3992-d92c-4460-8f3c-c6e5c42e03d1" (UID: "842b3992-d92c-4460-8f3c-c6e5c42e03d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:20:50.194327 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.194216    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842b3992-d92c-4460-8f3c-c6e5c42e03d1-kube-api-access-rzfjb" pod "842b3992-d92c-4460-8f3c-c6e5c42e03d1" (UID: "842b3992-d92c-4460-8f3c-c6e5c42e03d1"). InnerVolumeSpecName "kube-api-access-rzfjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:20:50.293883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.293848    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:50.293883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.293874    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842b3992-d92c-4460-8f3c-c6e5c42e03d1-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:50.293883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.293885    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzfjb\" (UniqueName: \"kubernetes.io/projected/842b3992-d92c-4460-8f3c-c6e5c42e03d1-kube-api-access-rzfjb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:50.955339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.955298    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b" event={"ID":"842b3992-d92c-4460-8f3c-c6e5c42e03d1","Type":"ContainerDied","Data":"aaee6abb26223e16ea5c5ca7af974ee7d9c9e41ed54afaa8f4c5a6344e1fb2a6"}
Nov 02 21:20:50.955339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.955340    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"
Nov 02 21:20:50.955585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:50.955356    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaee6abb26223e16ea5c5ca7af974ee7d9c9e41ed54afaa8f4c5a6344e1fb2a6"
Nov 02 21:20:51.109983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.109954    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:20:51.110403 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.110383    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:20:51.111041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.111023    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:20:51.170752 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.170677    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"]
Nov 02 21:20:51.442582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.442482    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk" podUID="9903f562-246b-4552-9d87-b03f1ace9b18" containerName="payload-processing" containerID="cri-o://659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb" gracePeriod=25
Nov 02 21:20:51.545739 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.545696    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq" podUID="b8f23572-02ac-4561-aa30-4c6ba90cd487" containerName="payload-pre-processing" containerID="cri-o://f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced" gracePeriod=25
Nov 02 21:20:51.607238 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:51.607207    4528 logging.go:55] [core] [Channel #221 SubChannel #223]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.46:50052", ServerName: "10.133.0.46:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.46:50052: connect: connection refused"
Nov 02 21:20:51.607238 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:51.607207    4528 logging.go:55] [core] [Channel #222 SubChannel #224]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.47:50052", ServerName: "10.133.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.47:50052: connect: connection refused"
Nov 02 21:20:51.709924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.709899    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:20:51.786343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.786320    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:20:51.808731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.808705    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9903f562-246b-4552-9d87-b03f1ace9b18-kube-api-access-rs26l\" (UniqueName: \"kubernetes.io/projected/9903f562-246b-4552-9d87-b03f1ace9b18-kube-api-access-rs26l\") pod \"9903f562-246b-4552-9d87-b03f1ace9b18\" (UID: \"9903f562-246b-4552-9d87-b03f1ace9b18\") "
Nov 02 21:20:51.808902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.808879    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9903f562-246b-4552-9d87-b03f1ace9b18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903f562-246b-4552-9d87-b03f1ace9b18-plugins-config-volume\") pod \"9903f562-246b-4552-9d87-b03f1ace9b18\" (UID: \"9903f562-246b-4552-9d87-b03f1ace9b18\") "
Nov 02 21:20:51.809273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.809228    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9903f562-246b-4552-9d87-b03f1ace9b18-plugins-config-volume" pod "9903f562-246b-4552-9d87-b03f1ace9b18" (UID: "9903f562-246b-4552-9d87-b03f1ace9b18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:20:51.810766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.810748    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9903f562-246b-4552-9d87-b03f1ace9b18-kube-api-access-rs26l" pod "9903f562-246b-4552-9d87-b03f1ace9b18" (UID: "9903f562-246b-4552-9d87-b03f1ace9b18"). InnerVolumeSpecName "kube-api-access-rs26l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:20:51.909789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.909760    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f23572-02ac-4561-aa30-4c6ba90cd487-kube-api-access-p546v\" (UniqueName: \"kubernetes.io/projected/b8f23572-02ac-4561-aa30-4c6ba90cd487-kube-api-access-p546v\") pod \"b8f23572-02ac-4561-aa30-4c6ba90cd487\" (UID: \"b8f23572-02ac-4561-aa30-4c6ba90cd487\") "
Nov 02 21:20:51.909952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.909877    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f23572-02ac-4561-aa30-4c6ba90cd487-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f23572-02ac-4561-aa30-4c6ba90cd487-plugins-config-volume\") pod \"b8f23572-02ac-4561-aa30-4c6ba90cd487\" (UID: \"b8f23572-02ac-4561-aa30-4c6ba90cd487\") "
Nov 02 21:20:51.910060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.910048    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9903f562-246b-4552-9d87-b03f1ace9b18-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:51.910102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.910063    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs26l\" (UniqueName: \"kubernetes.io/projected/9903f562-246b-4552-9d87-b03f1ace9b18-kube-api-access-rs26l\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:51.910282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.910255    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f23572-02ac-4561-aa30-4c6ba90cd487-plugins-config-volume" pod "b8f23572-02ac-4561-aa30-4c6ba90cd487" (UID: "b8f23572-02ac-4561-aa30-4c6ba90cd487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:20:51.911801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.911785    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f23572-02ac-4561-aa30-4c6ba90cd487-kube-api-access-p546v" pod "b8f23572-02ac-4561-aa30-4c6ba90cd487" (UID: "b8f23572-02ac-4561-aa30-4c6ba90cd487"). InnerVolumeSpecName "kube-api-access-p546v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:20:51.961225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.961147    4528 generic.go:356] "Generic (PLEG): container finished" podID="b8f23572-02ac-4561-aa30-4c6ba90cd487" containerID="f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced" exitCode=0
Nov 02 21:20:51.961225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.961194    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"
Nov 02 21:20:51.961424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.961224    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq" event={"ID":"b8f23572-02ac-4561-aa30-4c6ba90cd487","Type":"ContainerDied","Data":"f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced"}
Nov 02 21:20:51.961424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.961270    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq" event={"ID":"b8f23572-02ac-4561-aa30-4c6ba90cd487","Type":"ContainerDied","Data":"20783b270d2e99a6e9d73f4cf0a5de4aca5cbbbc0156490cb827f2fd35f110f2"}
Nov 02 21:20:51.961424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.961293    4528 scope.go:122] "RemoveContainer" containerID="f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced"
Nov 02 21:20:51.962582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.962561    4528 generic.go:356] "Generic (PLEG): container finished" podID="9903f562-246b-4552-9d87-b03f1ace9b18" containerID="659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb" exitCode=0
Nov 02 21:20:51.962695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.962606    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"
Nov 02 21:20:51.962695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.962641    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk" event={"ID":"9903f562-246b-4552-9d87-b03f1ace9b18","Type":"ContainerDied","Data":"659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb"}
Nov 02 21:20:51.962695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.962691    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk" event={"ID":"9903f562-246b-4552-9d87-b03f1ace9b18","Type":"ContainerDied","Data":"09d9ee44a23021a5b86220124b1ca18e51e7b4a70be2182f80ca8f4138d29d24"}
Nov 02 21:20:51.971531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.971515    4528 scope.go:122] "RemoveContainer" containerID="f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced"
Nov 02 21:20:51.971807 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:20:51.971781    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced\": container with ID starting with f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced not found: ID does not exist" containerID="f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced"
Nov 02 21:20:51.971921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.971814    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced"} err="failed to get container status \"f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced\": rpc error: code = NotFound desc = could not find container \"f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced\": container with ID starting with f366285c572aef5be6c1032b15fe52d1c96d134e9bcf6bf35ba3b848ba115ced not found: ID does not exist"
Nov 02 21:20:51.971921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.971835    4528 scope.go:122] "RemoveContainer" containerID="659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb"
Nov 02 21:20:51.979915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.979898    4528 scope.go:122] "RemoveContainer" containerID="659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb"
Nov 02 21:20:51.980183 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:20:51.980162    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb\": container with ID starting with 659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb not found: ID does not exist" containerID="659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb"
Nov 02 21:20:51.980246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:51.980190    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb"} err="failed to get container status \"659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb\": rpc error: code = NotFound desc = could not find container \"659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb\": container with ID starting with 659e50ab121e5fbf0ed880e5ccb11050f82ca375856fa16f77a56b5f718924cb not found: ID does not exist"
Nov 02 21:20:52.010602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.010566    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f23572-02ac-4561-aa30-4c6ba90cd487-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:52.010602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.010594    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p546v\" (UniqueName: \"kubernetes.io/projected/b8f23572-02ac-4561-aa30-4c6ba90cd487-kube-api-access-p546v\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:52.115878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.115826    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"]
Nov 02 21:20:52.118892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.118865    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk"]
Nov 02 21:20:52.215738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.215661    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"]
Nov 02 21:20:52.220189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.220165    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq"]
Nov 02 21:20:52.607620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.607568    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-170b4fea-5f9c9f6575-5m2lq" podUID="b8f23572-02ac-4561-aa30-4c6ba90cd487" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.46:50052\" within 1s: context deadline exceeded"
Nov 02 21:20:52.607832 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:52.607743    4528 logging.go:55] [core] [Channel #222 SubChannel #224]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.47:50052", ServerName: "10.133.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.47:50052: operation was canceled"
Nov 02 21:20:52.607832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.607582    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-170b4fea-b4bc7bc55-dnfqk" podUID="9903f562-246b-4552-9d87-b03f1ace9b18" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.47:50052\" within 1s: context deadline exceeded"
Nov 02 21:20:52.913212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.913140    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9903f562-246b-4552-9d87-b03f1ace9b18" path="/var/lib/kubelet/pods/9903f562-246b-4552-9d87-b03f1ace9b18/volumes"
Nov 02 21:20:52.913528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:52.913515    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f23572-02ac-4561-aa30-4c6ba90cd487" path="/var/lib/kubelet/pods/b8f23572-02ac-4561-aa30-4c6ba90cd487/volumes"
Nov 02 21:20:54.682754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.682717    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"]
Nov 02 21:20:54.683221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.683203    4528 cpu_manager.go:414] "containerMap: removing container" podUID="842b3992-d92c-4460-8f3c-c6e5c42e03d1" containerName="revoke-keys"
Nov 02 21:20:54.683271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.683223    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842b3992-d92c-4460-8f3c-c6e5c42e03d1" containerName="revoke-keys"
Nov 02 21:20:54.683271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.683258    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b8f23572-02ac-4561-aa30-4c6ba90cd487" containerName="payload-pre-processing"
Nov 02 21:20:54.683271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.683264    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f23572-02ac-4561-aa30-4c6ba90cd487" containerName="payload-pre-processing"
Nov 02 21:20:54.683365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.683276    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9903f562-246b-4552-9d87-b03f1ace9b18" containerName="payload-processing"
Nov 02 21:20:54.683365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.683281    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9903f562-246b-4552-9d87-b03f1ace9b18" containerName="payload-processing"
Nov 02 21:20:54.683365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.683340    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f23572-02ac-4561-aa30-4c6ba90cd487" containerName="payload-pre-processing"
Nov 02 21:20:54.683365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.683348    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9903f562-246b-4552-9d87-b03f1ace9b18" containerName="payload-processing"
Nov 02 21:20:54.683365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.683355    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="842b3992-d92c-4460-8f3c-c6e5c42e03d1" containerName="revoke-keys"
Nov 02 21:20:54.690838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.690818    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:54.740499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.740468    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"]
Nov 02 21:20:54.784322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.784286    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:20:54.784474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.784365    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:20:54.784474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.784414    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:20:54.788643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.788621    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:20:54.837355 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.837320    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f4fe932-csklx\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:54.837525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.837456    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f4fe932-csklx\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:54.837596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.837557    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lfbd\" (UniqueName: \"kubernetes.io/projected/151fa610-b799-4a65-baac-43a3584b4d37-kube-api-access-9lfbd\") pod \"maas-api-revoke-keys-e2e-ait-6f4fe932-csklx\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:54.938901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.938816    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f4fe932-csklx\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:54.938901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.938854    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f4fe932-csklx\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:54.939097 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.938908    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lfbd\" (UniqueName: \"kubernetes.io/projected/151fa610-b799-4a65-baac-43a3584b4d37-kube-api-access-9lfbd\") pod \"maas-api-revoke-keys-e2e-ait-6f4fe932-csklx\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:54.939554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.939530    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f4fe932-csklx\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:54.941311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.941285    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f4fe932-csklx\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:54.992522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:54.992494    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lfbd\" (UniqueName: \"kubernetes.io/projected/151fa610-b799-4a65-baac-43a3584b4d37-kube-api-access-9lfbd\") pod \"maas-api-revoke-keys-e2e-ait-6f4fe932-csklx\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:55.003321 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:55.003302    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:55.365427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:55.365404    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"]
Nov 02 21:20:55.367391 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:20:55.367359    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151fa610_b799_4a65_baac_43a3584b4d37.slice/crio-63e6f4a2639d997e97a4caae8f55ba6831b2ac34de47b1918d0ab6ba3cf3256b WatchSource:0}: Error finding container 63e6f4a2639d997e97a4caae8f55ba6831b2ac34de47b1918d0ab6ba3cf3256b: Status 404 returned error can't find the container with id 63e6f4a2639d997e97a4caae8f55ba6831b2ac34de47b1918d0ab6ba3cf3256b
Nov 02 21:20:55.984167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:55.984129    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx" event={"ID":"151fa610-b799-4a65-baac-43a3584b4d37","Type":"ContainerStarted","Data":"62c9a31c014dc0cb60a323cad64ce1f73f0f4dba7254713d60769f6fb573c6f2"}
Nov 02 21:20:55.984585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:55.984191    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx" event={"ID":"151fa610-b799-4a65-baac-43a3584b4d37","Type":"ContainerStarted","Data":"63e6f4a2639d997e97a4caae8f55ba6831b2ac34de47b1918d0ab6ba3cf3256b"}
Nov 02 21:20:56.094238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.094187    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx" podStartSLOduration=2.094159849 podStartE2EDuration="2.094159849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:56.094132005 +0000 UTC m=+743.357668323" watchObservedRunningTime="2026-11-02 21:20:56.094159849 +0000 UTC m=+743.357696167"
Nov 02 21:20:56.177066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.171230    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf" podUID="3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6" containerName="payload-pre-processing" containerID="cri-o://25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0" gracePeriod=25
Nov 02 21:20:56.431739 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.431713    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:20:56.554724 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.554638    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-plugins-config-volume\") pod \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\" (UID: \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\") "
Nov 02 21:20:56.554865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.554786    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-kube-api-access-p5p96\" (UniqueName: \"kubernetes.io/projected/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-kube-api-access-p5p96\") pod \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\" (UID: \"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6\") "
Nov 02 21:20:56.554990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.554971    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-plugins-config-volume" pod "3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6" (UID: "3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:20:56.555126 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.555100    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:56.556799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.556776    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-kube-api-access-p5p96" pod "3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6" (UID: "3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6"). InnerVolumeSpecName "kube-api-access-p5p96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:20:56.656165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.656130    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5p96\" (UniqueName: \"kubernetes.io/projected/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6-kube-api-access-p5p96\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:56.990870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.990841    4528 generic.go:356] "Generic (PLEG): container finished" podID="151fa610-b799-4a65-baac-43a3584b4d37" containerID="62c9a31c014dc0cb60a323cad64ce1f73f0f4dba7254713d60769f6fb573c6f2" exitCode=0
Nov 02 21:20:56.991259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.990930    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx" event={"ID":"151fa610-b799-4a65-baac-43a3584b4d37","Type":"ContainerDied","Data":"62c9a31c014dc0cb60a323cad64ce1f73f0f4dba7254713d60769f6fb573c6f2"}
Nov 02 21:20:56.992154 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.992135    4528 generic.go:356] "Generic (PLEG): container finished" podID="3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6" containerID="25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0" exitCode=0
Nov 02 21:20:56.992243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.992174    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"
Nov 02 21:20:56.992243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.992209    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf" event={"ID":"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6","Type":"ContainerDied","Data":"25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0"}
Nov 02 21:20:56.992243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.992241    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf" event={"ID":"3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6","Type":"ContainerDied","Data":"1e9a13880b817b5ad2fe7b77d4eed86b2b864d57a852568429a9ee1986033043"}
Nov 02 21:20:56.992355 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:56.992262    4528 scope.go:122] "RemoveContainer" containerID="25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0"
Nov 02 21:20:57.000691 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:57.000674    4528 scope.go:122] "RemoveContainer" containerID="25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0"
Nov 02 21:20:57.000958 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:20:57.000933    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0\": container with ID starting with 25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0 not found: ID does not exist" containerID="25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0"
Nov 02 21:20:57.001027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:57.000968    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0"} err="failed to get container status \"25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0\": rpc error: code = NotFound desc = could not find container \"25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0\": container with ID starting with 25fd3f0adb28ada1d839b74b1da52638d1bd72562b98abc13f6446cbe537d3c0 not found: ID does not exist"
Nov 02 21:20:57.205546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:57.205468    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"]
Nov 02 21:20:57.209326 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:57.209298    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5478da31-f77ddf987-twbbf"]
Nov 02 21:20:57.945623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:57.945589    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"]
Nov 02 21:20:58.041588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.041554    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"]
Nov 02 21:20:58.130197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.130176    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:20:58.287736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.275624    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-openshift-ca\") pod \"151fa610-b799-4a65-baac-43a3584b4d37\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") "
Nov 02 21:20:58.287736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.287310    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-xks-ca\") pod \"151fa610-b799-4a65-baac-43a3584b4d37\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") "
Nov 02 21:20:58.287736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.287386    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151fa610-b799-4a65-baac-43a3584b4d37-kube-api-access-9lfbd\" (UniqueName: \"kubernetes.io/projected/151fa610-b799-4a65-baac-43a3584b4d37-kube-api-access-9lfbd\") pod \"151fa610-b799-4a65-baac-43a3584b4d37\" (UID: \"151fa610-b799-4a65-baac-43a3584b4d37\") "
Nov 02 21:20:58.287736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.286545    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-openshift-ca" pod "151fa610-b799-4a65-baac-43a3584b4d37" (UID: "151fa610-b799-4a65-baac-43a3584b4d37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:20:58.287736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.287694    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:58.289777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.289752    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151fa610-b799-4a65-baac-43a3584b4d37-kube-api-access-9lfbd" pod "151fa610-b799-4a65-baac-43a3584b4d37" (UID: "151fa610-b799-4a65-baac-43a3584b4d37"). InnerVolumeSpecName "kube-api-access-9lfbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:20:58.293952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.293921    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-xks-ca" pod "151fa610-b799-4a65-baac-43a3584b4d37" (UID: "151fa610-b799-4a65-baac-43a3584b4d37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:20:58.388200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.388164    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lfbd\" (UniqueName: \"kubernetes.io/projected/151fa610-b799-4a65-baac-43a3584b4d37-kube-api-access-9lfbd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:58.388200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.388190    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fa610-b799-4a65-baac-43a3584b4d37-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:20:58.911893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:58.911862    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6" path="/var/lib/kubelet/pods/3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6/volumes"
Nov 02 21:20:59.008790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:59.007906    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx" event={"ID":"151fa610-b799-4a65-baac-43a3584b4d37","Type":"ContainerDied","Data":"63e6f4a2639d997e97a4caae8f55ba6831b2ac34de47b1918d0ab6ba3cf3256b"}
Nov 02 21:20:59.008790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:59.007975    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e6f4a2639d997e97a4caae8f55ba6831b2ac34de47b1918d0ab6ba3cf3256b"
Nov 02 21:20:59.008790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:20:59.008058    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"
Nov 02 21:21:00.714576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:00.714546    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"]
Nov 02 21:21:00.764328 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:00.764291    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-170b4fea-gjkqj"]
Nov 02 21:21:00.912308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:00.912272    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2ad085-4c35-4895-9c69-a48439905d1e" path="/var/lib/kubelet/pods/4a2ad085-4c35-4895-9c69-a48439905d1e/volumes"
Nov 02 21:21:01.165314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.165281    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"]
Nov 02 21:21:01.165706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.165694    4528 cpu_manager.go:414] "containerMap: removing container" podUID="151fa610-b799-4a65-baac-43a3584b4d37" containerName="revoke-keys"
Nov 02 21:21:01.165756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.165708    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151fa610-b799-4a65-baac-43a3584b4d37" containerName="revoke-keys"
Nov 02 21:21:01.165756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.165719    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6" containerName="payload-pre-processing"
Nov 02 21:21:01.165756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.165724    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6" containerName="payload-pre-processing"
Nov 02 21:21:01.165865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.165808    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="151fa610-b799-4a65-baac-43a3584b4d37" containerName="revoke-keys"
Nov 02 21:21:01.165865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.165817    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3c720c-c6d6-41aa-9fc8-7cbc388fdca6" containerName="payload-pre-processing"
Nov 02 21:21:01.170204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.170183    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.233820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.233777    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226ca5b9-618f-476e-a195-4d688632c554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-54a47302-7xm79\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.234024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.233847    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49s7q\" (UniqueName: \"kubernetes.io/projected/226ca5b9-618f-476e-a195-4d688632c554-kube-api-access-49s7q\") pod \"maas-api-revoke-keys-e2e-derive-54a47302-7xm79\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.234024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.233908    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226ca5b9-618f-476e-a195-4d688632c554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-54a47302-7xm79\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.265915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.265883    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:21:01.266128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.265957    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:21:01.266128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.265980    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:21:01.266128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.265985    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:21:01.269315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.269268    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"]
Nov 02 21:21:01.335081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.335048    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226ca5b9-618f-476e-a195-4d688632c554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-54a47302-7xm79\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.335282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.335092    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s7q\" (UniqueName: \"kubernetes.io/projected/226ca5b9-618f-476e-a195-4d688632c554-kube-api-access-49s7q\") pod \"maas-api-revoke-keys-e2e-derive-54a47302-7xm79\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.335282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.335156    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226ca5b9-618f-476e-a195-4d688632c554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-54a47302-7xm79\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.335762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.335743    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226ca5b9-618f-476e-a195-4d688632c554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-54a47302-7xm79\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.337363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.337339    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226ca5b9-618f-476e-a195-4d688632c554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-54a47302-7xm79\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.389668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.389599    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49s7q\" (UniqueName: \"kubernetes.io/projected/226ca5b9-618f-476e-a195-4d688632c554-kube-api-access-49s7q\") pod \"maas-api-revoke-keys-e2e-derive-54a47302-7xm79\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.483244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.483159    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:01.855098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:01.855070    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"]
Nov 02 21:21:01.857639 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:21:01.857602    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226ca5b9_618f_476e_a195_4d688632c554.slice/crio-03e8286b5cfb9aec509368d626dcaa6d32a1abf43b15e0286989401ada5e2dd8 WatchSource:0}: Error finding container 03e8286b5cfb9aec509368d626dcaa6d32a1abf43b15e0286989401ada5e2dd8: Status 404 returned error can't find the container with id 03e8286b5cfb9aec509368d626dcaa6d32a1abf43b15e0286989401ada5e2dd8
Nov 02 21:21:02.023781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:02.023751    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79" event={"ID":"226ca5b9-618f-476e-a195-4d688632c554","Type":"ContainerStarted","Data":"03e8286b5cfb9aec509368d626dcaa6d32a1abf43b15e0286989401ada5e2dd8"}
Nov 02 21:21:02.237708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:02.237679    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"]
Nov 02 21:21:02.240093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:02.240070    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5478da31-9mqzk"]
Nov 02 21:21:02.333384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:02.333353    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"]
Nov 02 21:21:02.333636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:02.333609    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy" containerID="cri-o://05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74" gracePeriod=30
Nov 02 21:21:02.912063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:02.912031    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd75759-e7c3-4950-82a0-3c6c4b174e87" path="/var/lib/kubelet/pods/bbd75759-e7c3-4950-82a0-3c6c4b174e87/volumes"
Nov 02 21:21:02.945966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:02.945885    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw" podUID="18abb932-9f6d-49e1-ab01-b1d9774e7a00" containerName="payload-processing" containerID="cri-o://f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff" gracePeriod=25
Nov 02 21:21:03.046237 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.046170    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb" podUID="d9762ede-8d11-4e06-a48e-f2f82e472d36" containerName="payload-pre-processing" containerID="cri-o://0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94" gracePeriod=25
Nov 02 21:21:03.046960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.046933    4528 generic.go:356] "Generic (PLEG): container finished" podID="226ca5b9-618f-476e-a195-4d688632c554" containerID="21315ba463eab2e9c4475c8cba03435831531d6cf30439340d4d339746369e51" exitCode=0
Nov 02 21:21:03.047064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.047043    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79" event={"ID":"226ca5b9-618f-476e-a195-4d688632c554","Type":"ContainerDied","Data":"21315ba463eab2e9c4475c8cba03435831531d6cf30439340d4d339746369e51"}
Nov 02 21:21:03.199767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.199705    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:21:03.254848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.254817    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18abb932-9f6d-49e1-ab01-b1d9774e7a00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18abb932-9f6d-49e1-ab01-b1d9774e7a00-plugins-config-volume\") pod \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\" (UID: \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\") "
Nov 02 21:21:03.255007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.254956    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18abb932-9f6d-49e1-ab01-b1d9774e7a00-kube-api-access-vlpl5\" (UniqueName: \"kubernetes.io/projected/18abb932-9f6d-49e1-ab01-b1d9774e7a00-kube-api-access-vlpl5\") pod \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\" (UID: \"18abb932-9f6d-49e1-ab01-b1d9774e7a00\") "
Nov 02 21:21:03.255298 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.255266    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18abb932-9f6d-49e1-ab01-b1d9774e7a00-plugins-config-volume" pod "18abb932-9f6d-49e1-ab01-b1d9774e7a00" (UID: "18abb932-9f6d-49e1-ab01-b1d9774e7a00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:21:03.256896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.256869    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18abb932-9f6d-49e1-ab01-b1d9774e7a00-kube-api-access-vlpl5" pod "18abb932-9f6d-49e1-ab01-b1d9774e7a00" (UID: "18abb932-9f6d-49e1-ab01-b1d9774e7a00"). InnerVolumeSpecName "kube-api-access-vlpl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:03.292843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.292819    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:21:03.355944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.355918    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9762ede-8d11-4e06-a48e-f2f82e472d36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9762ede-8d11-4e06-a48e-f2f82e472d36-plugins-config-volume\") pod \"d9762ede-8d11-4e06-a48e-f2f82e472d36\" (UID: \"d9762ede-8d11-4e06-a48e-f2f82e472d36\") "
Nov 02 21:21:03.356073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.355955    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9762ede-8d11-4e06-a48e-f2f82e472d36-kube-api-access-pmw2c\" (UniqueName: \"kubernetes.io/projected/d9762ede-8d11-4e06-a48e-f2f82e472d36-kube-api-access-pmw2c\") pod \"d9762ede-8d11-4e06-a48e-f2f82e472d36\" (UID: \"d9762ede-8d11-4e06-a48e-f2f82e472d36\") "
Nov 02 21:21:03.356235 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.356220    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18abb932-9f6d-49e1-ab01-b1d9774e7a00-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:03.356269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.356238    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlpl5\" (UniqueName: \"kubernetes.io/projected/18abb932-9f6d-49e1-ab01-b1d9774e7a00-kube-api-access-vlpl5\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:03.356313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.356294    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9762ede-8d11-4e06-a48e-f2f82e472d36-plugins-config-volume" pod "d9762ede-8d11-4e06-a48e-f2f82e472d36" (UID: "d9762ede-8d11-4e06-a48e-f2f82e472d36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:21:03.357993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.357974    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9762ede-8d11-4e06-a48e-f2f82e472d36-kube-api-access-pmw2c" pod "d9762ede-8d11-4e06-a48e-f2f82e472d36" (UID: "d9762ede-8d11-4e06-a48e-f2f82e472d36"). InnerVolumeSpecName "kube-api-access-pmw2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:03.457239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.457163    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9762ede-8d11-4e06-a48e-f2f82e472d36-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:03.457239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.457192    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmw2c\" (UniqueName: \"kubernetes.io/projected/d9762ede-8d11-4e06-a48e-f2f82e472d36-kube-api-access-pmw2c\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:03.654932 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.654900    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"]
Nov 02 21:21:03.655350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.655337    4528 cpu_manager.go:414] "containerMap: removing container" podUID="18abb932-9f6d-49e1-ab01-b1d9774e7a00" containerName="payload-processing"
Nov 02 21:21:03.655399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.655352    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18abb932-9f6d-49e1-ab01-b1d9774e7a00" containerName="payload-processing"
Nov 02 21:21:03.655399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.655383    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d9762ede-8d11-4e06-a48e-f2f82e472d36" containerName="payload-pre-processing"
Nov 02 21:21:03.655399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.655388    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9762ede-8d11-4e06-a48e-f2f82e472d36" containerName="payload-pre-processing"
Nov 02 21:21:03.655487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.655448    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9762ede-8d11-4e06-a48e-f2f82e472d36" containerName="payload-pre-processing"
Nov 02 21:21:03.655487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.655456    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="18abb932-9f6d-49e1-ab01-b1d9774e7a00" containerName="payload-processing"
Nov 02 21:21:03.658563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.658539    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.707358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.707281    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d6cb38-openshift-default-dockercfg-mbhbl\""
Nov 02 21:21:03.724277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.724239    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"]
Nov 02 21:21:03.760226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.760188    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.760226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.760226    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.760462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.760245    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.760462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.760287    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.760462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.760357    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.760462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.760380    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.760462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.760401    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4dw8\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-kube-api-access-q4dw8\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.760702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.760484    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.760702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.760521    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.861464 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.861429    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.861646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.861528    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.861646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.861563    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.861646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.861592    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.861646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.861618    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.861821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.861677    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.861821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.861715    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.861821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.861748    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4dw8\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-kube-api-access-q4dw8\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.861821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.861805    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.862298 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.862272    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.863262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.863226    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.863373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.863325    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.863586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.863565    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.863732 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.863709    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.866434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.865875    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.866434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.865885    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.916430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.916395    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"]
Nov 02 21:21:03.919152 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.919124    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4dw8\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-kube-api-access-q4dw8\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.919269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.919190    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:03.921728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.921708    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84de7f3a-dpz4b"]
Nov 02 21:21:03.971772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:03.971684    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:04.057726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.057697    4528 generic.go:356] "Generic (PLEG): container finished" podID="18abb932-9f6d-49e1-ab01-b1d9774e7a00" containerID="f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff" exitCode=0
Nov 02 21:21:04.058810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.057789    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw" event={"ID":"18abb932-9f6d-49e1-ab01-b1d9774e7a00","Type":"ContainerDied","Data":"f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff"}
Nov 02 21:21:04.058810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.057840    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"
Nov 02 21:21:04.058810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.057880    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw" event={"ID":"18abb932-9f6d-49e1-ab01-b1d9774e7a00","Type":"ContainerDied","Data":"0d0375454afd61f27e12d33f647e409e1fff6cdb520aac8e15e093827ab20fbc"}
Nov 02 21:21:04.058810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.057939    4528 scope.go:122] "RemoveContainer" containerID="f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff"
Nov 02 21:21:04.060498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.060476    4528 generic.go:356] "Generic (PLEG): container finished" podID="d9762ede-8d11-4e06-a48e-f2f82e472d36" containerID="0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94" exitCode=0
Nov 02 21:21:04.060620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.060533    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"
Nov 02 21:21:04.060620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.060542    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb" event={"ID":"d9762ede-8d11-4e06-a48e-f2f82e472d36","Type":"ContainerDied","Data":"0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94"}
Nov 02 21:21:04.060910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.060616    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb" event={"ID":"d9762ede-8d11-4e06-a48e-f2f82e472d36","Type":"ContainerDied","Data":"ed1e1dd08f75f44726e3988dcdb654fe5eb2889c0d196685dd09b324eb4aed1b"}
Nov 02 21:21:04.070321 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.070300    4528 scope.go:122] "RemoveContainer" containerID="f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff"
Nov 02 21:21:04.070649 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:04.070622    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff\": container with ID starting with f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff not found: ID does not exist" containerID="f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff"
Nov 02 21:21:04.070738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.070663    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff"} err="failed to get container status \"f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff\": rpc error: code = NotFound desc = could not find container \"f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff\": container with ID starting with f403c26e29db111190f76bb32f0c87e11109a0a3a91d297a187bcde04e19b2ff not found: ID does not exist"
Nov 02 21:21:04.070738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.070686    4528 scope.go:122] "RemoveContainer" containerID="0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94"
Nov 02 21:21:04.087908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.087398    4528 scope.go:122] "RemoveContainer" containerID="0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94"
Nov 02 21:21:04.087908 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:04.087751    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94\": container with ID starting with 0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94 not found: ID does not exist" containerID="0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94"
Nov 02 21:21:04.087908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.087783    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94"} err="failed to get container status \"0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94\": rpc error: code = NotFound desc = could not find container \"0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94\": container with ID starting with 0f4fe4a27af6ae1f444704cb081d7cf4727e9593259402bf8252e0e1003f6c94 not found: ID does not exist"
Nov 02 21:21:04.214473 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.214439    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"]
Nov 02 21:21:04.215924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.215899    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:04.217668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.217647    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84de7f3a-86c87f957f-2p2cb"]
Nov 02 21:21:04.265193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.265162    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/226ca5b9-618f-476e-a195-4d688632c554-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226ca5b9-618f-476e-a195-4d688632c554-maas-api-xks-ca\") pod \"226ca5b9-618f-476e-a195-4d688632c554\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") "
Nov 02 21:21:04.265356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.265217    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226ca5b9-618f-476e-a195-4d688632c554-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226ca5b9-618f-476e-a195-4d688632c554-maas-api-openshift-ca\") pod \"226ca5b9-618f-476e-a195-4d688632c554\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") "
Nov 02 21:21:04.265356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.265280    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226ca5b9-618f-476e-a195-4d688632c554-kube-api-access-49s7q\" (UniqueName: \"kubernetes.io/projected/226ca5b9-618f-476e-a195-4d688632c554-kube-api-access-49s7q\") pod \"226ca5b9-618f-476e-a195-4d688632c554\" (UID: \"226ca5b9-618f-476e-a195-4d688632c554\") "
Nov 02 21:21:04.265591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.265567    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226ca5b9-618f-476e-a195-4d688632c554-maas-api-openshift-ca" pod "226ca5b9-618f-476e-a195-4d688632c554" (UID: "226ca5b9-618f-476e-a195-4d688632c554"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:21:04.267238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.267220    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226ca5b9-618f-476e-a195-4d688632c554-maas-api-xks-ca" pod "226ca5b9-618f-476e-a195-4d688632c554" (UID: "226ca5b9-618f-476e-a195-4d688632c554"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:21:04.267293 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.267260    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226ca5b9-618f-476e-a195-4d688632c554-kube-api-access-49s7q" pod "226ca5b9-618f-476e-a195-4d688632c554" (UID: "226ca5b9-618f-476e-a195-4d688632c554"). InnerVolumeSpecName "kube-api-access-49s7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:04.318585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.318553    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"]
Nov 02 21:21:04.321814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.321789    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84de7f3a-56f89bd6b7-l7mvw"]
Nov 02 21:21:04.366455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.366423    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226ca5b9-618f-476e-a195-4d688632c554-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:04.366455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.366452    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226ca5b9-618f-476e-a195-4d688632c554-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:04.366455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.366462    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49s7q\" (UniqueName: \"kubernetes.io/projected/226ca5b9-618f-476e-a195-4d688632c554-kube-api-access-49s7q\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:04.397370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.397181    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"]
Nov 02 21:21:04.402017 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:21:04.401976    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b41774c_4f2c_4ba3_bae3_36b4e1b0150c.slice/crio-8d246825164134e68e53b1184301e47c20780987158156da6dc484e09ca512db WatchSource:0}: Error finding container 8d246825164134e68e53b1184301e47c20780987158156da6dc484e09ca512db: Status 404 returned error can't find the container with id 8d246825164134e68e53b1184301e47c20780987158156da6dc484e09ca512db
Nov 02 21:21:04.404528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.404495    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:04.404616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.404565    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:04.404665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.404624    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:04.912973 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.912923    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18abb932-9f6d-49e1-ab01-b1d9774e7a00" path="/var/lib/kubelet/pods/18abb932-9f6d-49e1-ab01-b1d9774e7a00/volumes"
Nov 02 21:21:04.913359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.913346    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842b3992-d92c-4460-8f3c-c6e5c42e03d1" path="/var/lib/kubelet/pods/842b3992-d92c-4460-8f3c-c6e5c42e03d1/volumes"
Nov 02 21:21:04.913662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:04.913651    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9762ede-8d11-4e06-a48e-f2f82e472d36" path="/var/lib/kubelet/pods/d9762ede-8d11-4e06-a48e-f2f82e472d36/volumes"
Nov 02 21:21:05.066763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:05.066726    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" event={"ID":"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c","Type":"ContainerStarted","Data":"f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f"}
Nov 02 21:21:05.067217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:05.066792    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" event={"ID":"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c","Type":"ContainerStarted","Data":"8d246825164134e68e53b1184301e47c20780987158156da6dc484e09ca512db"}
Nov 02 21:21:05.068075 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:05.068056    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"
Nov 02 21:21:05.068226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:05.068091    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79" event={"ID":"226ca5b9-618f-476e-a195-4d688632c554","Type":"ContainerDied","Data":"03e8286b5cfb9aec509368d626dcaa6d32a1abf43b15e0286989401ada5e2dd8"}
Nov 02 21:21:05.068226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:05.068150    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03e8286b5cfb9aec509368d626dcaa6d32a1abf43b15e0286989401ada5e2dd8"
Nov 02 21:21:05.179906 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:05.179806    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" podStartSLOduration=2.179792475 podStartE2EDuration="2.179792475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:05.178105393 +0000 UTC m=+752.441641712" watchObservedRunningTime="2026-11-02 21:21:05.179792475 +0000 UTC m=+752.443328876"
Nov 02 21:21:05.578990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:05.578957    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:21:05.972764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:05.972676    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:05.978834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:05.978808    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:06.076720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:06.076690    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:06.077591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:06.077571    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:06.136631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:06.136602    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"]
Nov 02 21:21:06.213840 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:06.213805    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"]
Nov 02 21:21:07.611150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.611083    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:21:07.703094 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703061    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-kube-api-access-zxbdr\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-kube-api-access-zxbdr\") pod \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") "
Nov 02 21:21:07.703283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703126    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-token\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-token\") pod \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") "
Nov 02 21:21:07.703283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703161    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-envoy\") pod \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") "
Nov 02 21:21:07.703283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703200    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-socket\") pod \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") "
Nov 02 21:21:07.703283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703232    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-credential-socket\") pod \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") "
Nov 02 21:21:07.703591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703285    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-podinfo\") pod \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") "
Nov 02 21:21:07.703591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703382    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-certs\") pod \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") "
Nov 02 21:21:07.703591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703410    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-data\") pod \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") "
Nov 02 21:21:07.703591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703474    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istiod-ca-cert\") pod \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\" (UID: \"76d007aa-3135-4dbb-b99a-cfff7a7b64c5\") "
Nov 02 21:21:07.703591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703537    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-credential-socket" pod "76d007aa-3135-4dbb-b99a-cfff7a7b64c5" (UID: "76d007aa-3135-4dbb-b99a-cfff7a7b64c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:07.703591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703550    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-socket" pod "76d007aa-3135-4dbb-b99a-cfff7a7b64c5" (UID: "76d007aa-3135-4dbb-b99a-cfff7a7b64c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:07.703888 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703754    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-certs" pod "76d007aa-3135-4dbb-b99a-cfff7a7b64c5" (UID: "76d007aa-3135-4dbb-b99a-cfff7a7b64c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:07.703888 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703872    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-data" pod "76d007aa-3135-4dbb-b99a-cfff7a7b64c5" (UID: "76d007aa-3135-4dbb-b99a-cfff7a7b64c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:07.703995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703877    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istiod-ca-cert" pod "76d007aa-3135-4dbb-b99a-cfff7a7b64c5" (UID: "76d007aa-3135-4dbb-b99a-cfff7a7b64c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:21:07.703995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703908    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:07.703995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703927    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:07.703995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.703941    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:07.705435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.705412    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-envoy" pod "76d007aa-3135-4dbb-b99a-cfff7a7b64c5" (UID: "76d007aa-3135-4dbb-b99a-cfff7a7b64c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:07.705435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.705427    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-token" pod "76d007aa-3135-4dbb-b99a-cfff7a7b64c5" (UID: "76d007aa-3135-4dbb-b99a-cfff7a7b64c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:07.705590 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.705413    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-kube-api-access-zxbdr" pod "76d007aa-3135-4dbb-b99a-cfff7a7b64c5" (UID: "76d007aa-3135-4dbb-b99a-cfff7a7b64c5"). InnerVolumeSpecName "kube-api-access-zxbdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:07.705590 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.705559    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-podinfo" pod "76d007aa-3135-4dbb-b99a-cfff7a7b64c5" (UID: "76d007aa-3135-4dbb-b99a-cfff7a7b64c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:21:07.804616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.804579    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:07.804616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.804608    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:07.804616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.804617    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:07.804842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.804628    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxbdr\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-kube-api-access-zxbdr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:07.804842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.804638    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:07.804842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:07.804650    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d007aa-3135-4dbb-b99a-cfff7a7b64c5-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:08.088998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.088967    4528 generic.go:356] "Generic (PLEG): container finished" podID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerID="05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74" exitCode=0
Nov 02 21:21:08.089213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.089019    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"
Nov 02 21:21:08.089213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.089057    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" event={"ID":"76d007aa-3135-4dbb-b99a-cfff7a7b64c5","Type":"ContainerDied","Data":"05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74"}
Nov 02 21:21:08.089213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.089134    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7" event={"ID":"76d007aa-3135-4dbb-b99a-cfff7a7b64c5","Type":"ContainerDied","Data":"5353a67cbe43c3cb16adbb5f0f7dc9d60a8dcb9343318634b6351d44236c40dc"}
Nov 02 21:21:08.089213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.089165    4528 scope.go:122] "RemoveContainer" containerID="05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74"
Nov 02 21:21:08.098091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.098077    4528 scope.go:122] "RemoveContainer" containerID="3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942"
Nov 02 21:21:08.105946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.105930    4528 scope.go:122] "RemoveContainer" containerID="05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74"
Nov 02 21:21:08.106236 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:08.106218    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74\": container with ID starting with 05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74 not found: ID does not exist" containerID="05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74"
Nov 02 21:21:08.106315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.106245    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74"} err="failed to get container status \"05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74\": rpc error: code = NotFound desc = could not find container \"05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74\": container with ID starting with 05df56589019870dfee370e1ed94d26533f6cb9e5ffe7bedcb70abff0212fa74 not found: ID does not exist"
Nov 02 21:21:08.106315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.106261    4528 scope.go:122] "RemoveContainer" containerID="3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942"
Nov 02 21:21:08.106503 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:08.106483    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942\": container with ID starting with 3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942 not found: ID does not exist" containerID="3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942"
Nov 02 21:21:08.106544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.106509    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942"} err="failed to get container status \"3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942\": rpc error: code = NotFound desc = could not find container \"3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942\": container with ID starting with 3efc4c573305a60067f7308bc4cfde30a36df02649f6c1c5df3591ae7e5c1942 not found: ID does not exist"
Nov 02 21:21:08.248169 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.248140    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"]
Nov 02 21:21:08.295357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.295325    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5478da31-gw-openshift-default-58c8dcb85-qhhk7"]
Nov 02 21:21:08.913287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:08.913256    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" path="/var/lib/kubelet/pods/76d007aa-3135-4dbb-b99a-cfff7a7b64c5/volumes"
Nov 02 21:21:11.137745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.137681    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp" podUID="9dcb42db-eb12-4bd6-a0f4-39025f99377d" containerName="payload-processing" containerID="cri-o://16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6" gracePeriod=25
Nov 02 21:21:11.219129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.218808    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5" podUID="4f366747-a317-4e69-b600-37643e5cc23a" containerName="payload-pre-processing" containerID="cri-o://5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863" gracePeriod=25
Nov 02 21:21:11.430288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.429144    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:21:11.526599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.526575    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"]
Nov 02 21:21:11.526948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.526935    4528 cpu_manager.go:414] "containerMap: removing container" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy"
Nov 02 21:21:11.527001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.526950    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy"
Nov 02 21:21:11.527001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.526969    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9dcb42db-eb12-4bd6-a0f4-39025f99377d" containerName="payload-processing"
Nov 02 21:21:11.527001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.526974    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcb42db-eb12-4bd6-a0f4-39025f99377d" containerName="payload-processing"
Nov 02 21:21:11.527001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.526981    4528 cpu_manager.go:414] "containerMap: removing container" podUID="226ca5b9-618f-476e-a195-4d688632c554" containerName="revoke-keys"
Nov 02 21:21:11.527001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.526987    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226ca5b9-618f-476e-a195-4d688632c554" containerName="revoke-keys"
Nov 02 21:21:11.527001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.526997    4528 cpu_manager.go:414] "containerMap: removing container" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy"
Nov 02 21:21:11.527001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.527002    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy"
Nov 02 21:21:11.527302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.527080    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy"
Nov 02 21:21:11.527302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.527088    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcb42db-eb12-4bd6-a0f4-39025f99377d" containerName="payload-processing"
Nov 02 21:21:11.527302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.527094    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="226ca5b9-618f-476e-a195-4d688632c554" containerName="revoke-keys"
Nov 02 21:21:11.530289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.530271    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:11.533629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.533600    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:21:11.543159 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.543140    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcb42db-eb12-4bd6-a0f4-39025f99377d-kube-api-access-jhpkd\" (UniqueName: \"kubernetes.io/projected/9dcb42db-eb12-4bd6-a0f4-39025f99377d-kube-api-access-jhpkd\") pod \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\" (UID: \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\") "
Nov 02 21:21:11.543248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.543184    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dcb42db-eb12-4bd6-a0f4-39025f99377d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcb42db-eb12-4bd6-a0f4-39025f99377d-plugins-config-volume\") pod \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\" (UID: \"9dcb42db-eb12-4bd6-a0f4-39025f99377d\") "
Nov 02 21:21:11.543574 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.543554    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dcb42db-eb12-4bd6-a0f4-39025f99377d-plugins-config-volume" pod "9dcb42db-eb12-4bd6-a0f4-39025f99377d" (UID: "9dcb42db-eb12-4bd6-a0f4-39025f99377d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:21:11.545248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.545227    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcb42db-eb12-4bd6-a0f4-39025f99377d-kube-api-access-jhpkd" pod "9dcb42db-eb12-4bd6-a0f4-39025f99377d" (UID: "9dcb42db-eb12-4bd6-a0f4-39025f99377d"). InnerVolumeSpecName "kube-api-access-jhpkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:11.570888 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.570859    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"]
Nov 02 21:21:11.583728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.583708    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f51151\""
Nov 02 21:21:11.600066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.600041    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f51151-dockercfg-dm9gr\""
Nov 02 21:21:11.644136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.644096    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f366747-a317-4e69-b600-37643e5cc23a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f366747-a317-4e69-b600-37643e5cc23a-plugins-config-volume\") pod \"4f366747-a317-4e69-b600-37643e5cc23a\" (UID: \"4f366747-a317-4e69-b600-37643e5cc23a\") "
Nov 02 21:21:11.644293 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.644260    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f366747-a317-4e69-b600-37643e5cc23a-kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/4f366747-a317-4e69-b600-37643e5cc23a-kube-api-access-k6chk\") pod \"4f366747-a317-4e69-b600-37643e5cc23a\" (UID: \"4f366747-a317-4e69-b600-37643e5cc23a\") "
Nov 02 21:21:11.644417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.644401    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2\" (UID: \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:11.644486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.644431    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsksj\" (UniqueName: \"kubernetes.io/projected/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-kube-api-access-xsksj\") pod \"payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2\" (UID: \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:11.644486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.644449    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f366747-a317-4e69-b600-37643e5cc23a-plugins-config-volume" pod "4f366747-a317-4e69-b600-37643e5cc23a" (UID: "4f366747-a317-4e69-b600-37643e5cc23a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:21:11.644630 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.644616    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhpkd\" (UniqueName: \"kubernetes.io/projected/9dcb42db-eb12-4bd6-a0f4-39025f99377d-kube-api-access-jhpkd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:11.644675 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.644633    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dcb42db-eb12-4bd6-a0f4-39025f99377d-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:11.644675 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.644642    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f366747-a317-4e69-b600-37643e5cc23a-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:11.646255 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.646237    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f366747-a317-4e69-b600-37643e5cc23a-kube-api-access-k6chk" pod "4f366747-a317-4e69-b600-37643e5cc23a" (UID: "4f366747-a317-4e69-b600-37643e5cc23a"). InnerVolumeSpecName "kube-api-access-k6chk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:11.745721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.745637    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2\" (UID: \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:11.745721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.745673    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsksj\" (UniqueName: \"kubernetes.io/projected/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-kube-api-access-xsksj\") pod \"payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2\" (UID: \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:11.745911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.745759    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/4f366747-a317-4e69-b600-37643e5cc23a-kube-api-access-k6chk\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:11.746423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.746402    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2\" (UID: \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:11.798646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.798614    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsksj\" (UniqueName: \"kubernetes.io/projected/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-kube-api-access-xsksj\") pod \"payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2\" (UID: \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:11.843485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:11.843455    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:12.110075 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.110044    4528 generic.go:356] "Generic (PLEG): container finished" podID="4f366747-a317-4e69-b600-37643e5cc23a" containerID="5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863" exitCode=0
Nov 02 21:21:12.110272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.110095    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"
Nov 02 21:21:12.110272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.110153    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5" event={"ID":"4f366747-a317-4e69-b600-37643e5cc23a","Type":"ContainerDied","Data":"5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863"}
Nov 02 21:21:12.110272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.110228    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5" event={"ID":"4f366747-a317-4e69-b600-37643e5cc23a","Type":"ContainerDied","Data":"406d2825d041a44a56c77e1bf9037d1d43c413ba91cb7ac677793a059183ecb1"}
Nov 02 21:21:12.110272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.110253    4528 scope.go:122] "RemoveContainer" containerID="5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863"
Nov 02 21:21:12.111380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.111359    4528 generic.go:356] "Generic (PLEG): container finished" podID="9dcb42db-eb12-4bd6-a0f4-39025f99377d" containerID="16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6" exitCode=0
Nov 02 21:21:12.111471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.111403    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"
Nov 02 21:21:12.111471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.111413    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp" event={"ID":"9dcb42db-eb12-4bd6-a0f4-39025f99377d","Type":"ContainerDied","Data":"16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6"}
Nov 02 21:21:12.111471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.111445    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp" event={"ID":"9dcb42db-eb12-4bd6-a0f4-39025f99377d","Type":"ContainerDied","Data":"8f16a9098e1e024caa3736232bfe2b172c48dc46f35c945e74b48641b35e4896"}
Nov 02 21:21:12.118981 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.118865    4528 scope.go:122] "RemoveContainer" containerID="5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863"
Nov 02 21:21:12.119275 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:12.119249    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863\": container with ID starting with 5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863 not found: ID does not exist" containerID="5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863"
Nov 02 21:21:12.119397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.119283    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863"} err="failed to get container status \"5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863\": rpc error: code = NotFound desc = could not find container \"5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863\": container with ID starting with 5085d1744dc316b41e9acb433cdd5b1c12d3fe32097de78894cf4c6d8762b863 not found: ID does not exist"
Nov 02 21:21:12.119397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.119307    4528 scope.go:122] "RemoveContainer" containerID="16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6"
Nov 02 21:21:12.127426 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.127409    4528 scope.go:122] "RemoveContainer" containerID="16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6"
Nov 02 21:21:12.127653 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:12.127637    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6\": container with ID starting with 16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6 not found: ID does not exist" containerID="16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6"
Nov 02 21:21:12.127698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.127659    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6"} err="failed to get container status \"16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6\": rpc error: code = NotFound desc = could not find container \"16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6\": container with ID starting with 16bbac448a60edc06b4508c1e16e469b469bc1cb6a7e5313524a208c795517c6 not found: ID does not exist"
Nov 02 21:21:12.207003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.206981    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"]
Nov 02 21:21:12.209647 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:21:12.209617    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc82c18_b2a2_47cc_bfbe_4f48c80a5cb0.slice/crio-c5d4c01db1d280ef50bc28baa83255da3fc2e4f018d654c1f894655fe9d2619a WatchSource:0}: Error finding container c5d4c01db1d280ef50bc28baa83255da3fc2e4f018d654c1f894655fe9d2619a: Status 404 returned error can't find the container with id c5d4c01db1d280ef50bc28baa83255da3fc2e4f018d654c1f894655fe9d2619a
Nov 02 21:21:12.261850 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.261821    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"]
Nov 02 21:21:12.268332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.268311    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6f4fe932-6fff5bf6d7-lmkp5"]
Nov 02 21:21:12.374730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.374598    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"]
Nov 02 21:21:12.377486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.377454    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f4fe932-7f6448cb57-qf4gp"]
Nov 02 21:21:12.912030 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.912001    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f366747-a317-4e69-b600-37643e5cc23a" path="/var/lib/kubelet/pods/4f366747-a317-4e69-b600-37643e5cc23a/volumes"
Nov 02 21:21:12.912360 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:12.912347    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dcb42db-eb12-4bd6-a0f4-39025f99377d" path="/var/lib/kubelet/pods/9dcb42db-eb12-4bd6-a0f4-39025f99377d/volumes"
Nov 02 21:21:13.119507 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:13.119477    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2" event={"ID":"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0","Type":"ContainerStarted","Data":"efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589"}
Nov 02 21:21:13.119712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:13.119548    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2" event={"ID":"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0","Type":"ContainerStarted","Data":"c5d4c01db1d280ef50bc28baa83255da3fc2e4f018d654c1f894655fe9d2619a"}
Nov 02 21:21:13.119712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:13.119576    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:13.229248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:13.229146    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2" podStartSLOduration=2.229132309 podStartE2EDuration="2.229132309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:13.226842225 +0000 UTC m=+760.490378541" watchObservedRunningTime="2026-11-02 21:21:13.229132309 +0000 UTC m=+760.492668629"
Nov 02 21:21:14.112097 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:14.112056    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:21:18.436161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:18.436124    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"]
Nov 02 21:21:18.483313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:18.483274    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f4fe932-csklx"]
Nov 02 21:21:18.911350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:18.911322    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151fa610-b799-4a65-baac-43a3584b4d37" path="/var/lib/kubelet/pods/151fa610-b799-4a65-baac-43a3584b4d37/volumes"
Nov 02 21:21:19.744534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.744503    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"]
Nov 02 21:21:19.745122 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.745087    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4f366747-a317-4e69-b600-37643e5cc23a" containerName="payload-pre-processing"
Nov 02 21:21:19.745216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.745123    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f366747-a317-4e69-b600-37643e5cc23a" containerName="payload-pre-processing"
Nov 02 21:21:19.745279 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.745254    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f366747-a317-4e69-b600-37643e5cc23a" containerName="payload-pre-processing"
Nov 02 21:21:19.745279 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.745268    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d007aa-3135-4dbb-b99a-cfff7a7b64c5" containerName="istio-proxy"
Nov 02 21:21:19.747987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.747968    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.820853 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.820814    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6fa533-openshift-default-dockercfg-psqwg\""
Nov 02 21:21:19.850614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.850581    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.850614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.850620    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbtm\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-kube-api-access-kfbtm\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.850854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.850694    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.850854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.850724    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.850854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.850797    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.850854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.850819    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.851045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.850863    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.851045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.850881    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.851045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.850990    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.865472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.865437    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"]
Nov 02 21:21:19.951719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.951690    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.951719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.951721    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.951942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.951753    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.951942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.951776    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.951942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.951794    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbtm\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-kube-api-access-kfbtm\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.951942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.951819    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.951942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.951843    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.951942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.951885    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.951942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.951908    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.952284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.952168    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.952387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.952360    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.952517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.952425    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.952517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.952478    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.952713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.952692    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.954102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.954081    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:19.954511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:19.954493    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:20.013057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:20.013028    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:20.018090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:20.018023    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbtm\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-kube-api-access-kfbtm\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:20.060208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:20.060176    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:20.473432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:20.473347    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"]
Nov 02 21:21:20.475654 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:21:20.475625    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af8bacd_5cbc_4c50_97a4_0fdf084d4bd9.slice/crio-d6413ae2316f908e41fe774718ab5b41f6d2403da79c36baa498ed92c432154e WatchSource:0}: Error finding container d6413ae2316f908e41fe774718ab5b41f6d2403da79c36baa498ed92c432154e: Status 404 returned error can't find the container with id d6413ae2316f908e41fe774718ab5b41f6d2403da79c36baa498ed92c432154e
Nov 02 21:21:20.477938 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:20.477905    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:20.478025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:20.477982    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:20.478081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:20.478026    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:21.160896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:21.160864    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" event={"ID":"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9","Type":"ContainerStarted","Data":"b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf"}
Nov 02 21:21:21.161312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:21.160924    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" event={"ID":"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9","Type":"ContainerStarted","Data":"d6413ae2316f908e41fe774718ab5b41f6d2403da79c36baa498ed92c432154e"}
Nov 02 21:21:21.365401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:21.365341    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" podStartSLOduration=2.365325005 podStartE2EDuration="2.365325005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:21.317547638 +0000 UTC m=+768.581083958" watchObservedRunningTime="2026-11-02 21:21:21.365325005 +0000 UTC m=+768.628861323"
Nov 02 21:21:21.498695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:21.498613    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"]
Nov 02 21:21:21.501077 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:21.501052    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-54a47302-7xm79"]
Nov 02 21:21:21.561775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:21.561733    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"]
Nov 02 21:21:21.563052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:21.563015    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy" containerID="cri-o://457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36" gracePeriod=30
Nov 02 21:21:21.784525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:21.784497    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:21:22.060743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:22.060668    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:22.065622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:22.065597    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:22.165976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:22.165945    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:22.166888 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:22.166867    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:21:22.911804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:22.911771    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226ca5b9-618f-476e-a195-4d688632c554" path="/var/lib/kubelet/pods/226ca5b9-618f-476e-a195-4d688632c554/volumes"
Nov 02 21:21:24.129054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:24.129022    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 21:21:26.811737 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.811713    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:21:26.940707 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.940679    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-certs\") pod \"75499c01-f0db-48cf-b2de-33e2fff4502b\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") "
Nov 02 21:21:26.940867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.940720    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-kube-api-access-bq6v8\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-kube-api-access-bq6v8\") pod \"75499c01-f0db-48cf-b2de-33e2fff4502b\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") "
Nov 02 21:21:26.940867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.940762    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-data\") pod \"75499c01-f0db-48cf-b2de-33e2fff4502b\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") "
Nov 02 21:21:26.940867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.940787    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-podinfo\") pod \"75499c01-f0db-48cf-b2de-33e2fff4502b\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") "
Nov 02 21:21:26.940867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.940806    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-socket\") pod \"75499c01-f0db-48cf-b2de-33e2fff4502b\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") "
Nov 02 21:21:26.940867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.940834    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-credential-socket\") pod \"75499c01-f0db-48cf-b2de-33e2fff4502b\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") "
Nov 02 21:21:26.941185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.940968    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75499c01-f0db-48cf-b2de-33e2fff4502b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75499c01-f0db-48cf-b2de-33e2fff4502b-istiod-ca-cert\") pod \"75499c01-f0db-48cf-b2de-33e2fff4502b\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") "
Nov 02 21:21:26.941185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.940995    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-token\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-token\") pod \"75499c01-f0db-48cf-b2de-33e2fff4502b\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") "
Nov 02 21:21:26.941185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.941020    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-certs" pod "75499c01-f0db-48cf-b2de-33e2fff4502b" (UID: "75499c01-f0db-48cf-b2de-33e2fff4502b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:26.941185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.941046    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-envoy\") pod \"75499c01-f0db-48cf-b2de-33e2fff4502b\" (UID: \"75499c01-f0db-48cf-b2de-33e2fff4502b\") "
Nov 02 21:21:26.941185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.941092    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-data" pod "75499c01-f0db-48cf-b2de-33e2fff4502b" (UID: "75499c01-f0db-48cf-b2de-33e2fff4502b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:26.941185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.941170    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-socket" pod "75499c01-f0db-48cf-b2de-33e2fff4502b" (UID: "75499c01-f0db-48cf-b2de-33e2fff4502b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:26.941496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.941464    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:26.941496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.941486    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:26.941606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.941500    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:26.941606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.941528    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-credential-socket" pod "75499c01-f0db-48cf-b2de-33e2fff4502b" (UID: "75499c01-f0db-48cf-b2de-33e2fff4502b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:26.941712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.941607    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75499c01-f0db-48cf-b2de-33e2fff4502b-istiod-ca-cert" pod "75499c01-f0db-48cf-b2de-33e2fff4502b" (UID: "75499c01-f0db-48cf-b2de-33e2fff4502b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:21:26.943288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.943258    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-envoy" pod "75499c01-f0db-48cf-b2de-33e2fff4502b" (UID: "75499c01-f0db-48cf-b2de-33e2fff4502b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:21:26.943397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.943329    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-token" pod "75499c01-f0db-48cf-b2de-33e2fff4502b" (UID: "75499c01-f0db-48cf-b2de-33e2fff4502b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:26.943520 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.943497    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-podinfo" pod "75499c01-f0db-48cf-b2de-33e2fff4502b" (UID: "75499c01-f0db-48cf-b2de-33e2fff4502b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:21:26.943611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:26.943526    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-kube-api-access-bq6v8" pod "75499c01-f0db-48cf-b2de-33e2fff4502b" (UID: "75499c01-f0db-48cf-b2de-33e2fff4502b"). InnerVolumeSpecName "kube-api-access-bq6v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:21:27.042539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.042505    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:27.042539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.042532    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:27.042539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.042542    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75499c01-f0db-48cf-b2de-33e2fff4502b-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:27.042539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.042550    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:27.042814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.042560    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75499c01-f0db-48cf-b2de-33e2fff4502b-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:27.042814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.042570    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6v8\" (UniqueName: \"kubernetes.io/projected/75499c01-f0db-48cf-b2de-33e2fff4502b-kube-api-access-bq6v8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:21:27.194247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.194143    4528 generic.go:356] "Generic (PLEG): container finished" podID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerID="457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36" exitCode=0
Nov 02 21:21:27.194247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.194208    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"
Nov 02 21:21:27.194247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.194223    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" event={"ID":"75499c01-f0db-48cf-b2de-33e2fff4502b","Type":"ContainerDied","Data":"457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36"}
Nov 02 21:21:27.194471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.194276    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r" event={"ID":"75499c01-f0db-48cf-b2de-33e2fff4502b","Type":"ContainerDied","Data":"3d68e0970ac04308f775c1017c9c40211b1bf134ce156d67a44c65ecfcd25570"}
Nov 02 21:21:27.194471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.194297    4528 scope.go:122] "RemoveContainer" containerID="457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36"
Nov 02 21:21:27.203713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.203697    4528 scope.go:122] "RemoveContainer" containerID="8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357"
Nov 02 21:21:27.211761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.211741    4528 scope.go:122] "RemoveContainer" containerID="457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36"
Nov 02 21:21:27.212058 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:27.212029    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36\": container with ID starting with 457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36 not found: ID does not exist" containerID="457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36"
Nov 02 21:21:27.212180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.212070    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36"} err="failed to get container status \"457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36\": rpc error: code = NotFound desc = could not find container \"457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36\": container with ID starting with 457c96e7741821fae4ccf35636b37ad7c803a16b4e871a6f41d1c5810aa52a36 not found: ID does not exist"
Nov 02 21:21:27.212180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.212094    4528 scope.go:122] "RemoveContainer" containerID="8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357"
Nov 02 21:21:27.212404 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:27.212387    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357\": container with ID starting with 8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357 not found: ID does not exist" containerID="8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357"
Nov 02 21:21:27.212455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.212411    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357"} err="failed to get container status \"8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357\": rpc error: code = NotFound desc = could not find container \"8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357\": container with ID starting with 8f83be6aa413f6c6906b05b9531b70a06f0fe36f8f0405470e949e58a5879357 not found: ID does not exist"
Nov 02 21:21:27.347047 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.347018    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"]
Nov 02 21:21:27.393883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:27.393856    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-54a47302-openshift-default-5555c5f6fc-wzk5r"]
Nov 02 21:21:28.912262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:28.912228    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" path="/var/lib/kubelet/pods/75499c01-f0db-48cf-b2de-33e2fff4502b/volumes"
Nov 02 21:21:30.872712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.872679    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"]
Nov 02 21:21:30.873141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.873067    4528 cpu_manager.go:414] "containerMap: removing container" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy"
Nov 02 21:21:30.873141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.873079    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy"
Nov 02 21:21:30.873141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.873124    4528 cpu_manager.go:414] "containerMap: removing container" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy"
Nov 02 21:21:30.873141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.873134    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy"
Nov 02 21:21:30.873277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.873194    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy"
Nov 02 21:21:30.873277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.873203    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="75499c01-f0db-48cf-b2de-33e2fff4502b" containerName="istio-proxy"
Nov 02 21:21:30.878120 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.878094    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:30.931862 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.931818    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"]
Nov 02 21:21:30.975422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.975391    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-6fa533-dockercfg-mq2j7\""
Nov 02 21:21:30.975733 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.975713    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-6fa533\""
Nov 02 21:21:30.988461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.988421    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twvcp\" (UniqueName: \"kubernetes.io/projected/e9e8e764-309c-4788-8268-b27f2dc8f290-kube-api-access-twvcp\") pod \"payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7\" (UID: \"e9e8e764-309c-4788-8268-b27f2dc8f290\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:30.988739 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:30.988711    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e8e764-309c-4788-8268-b27f2dc8f290-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7\" (UID: \"e9e8e764-309c-4788-8268-b27f2dc8f290\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:31.053105 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.053068    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"]
Nov 02 21:21:31.059436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.059405    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:31.090313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.090279    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e8e764-309c-4788-8268-b27f2dc8f290-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7\" (UID: \"e9e8e764-309c-4788-8268-b27f2dc8f290\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:31.090497 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.090408    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twvcp\" (UniqueName: \"kubernetes.io/projected/e9e8e764-309c-4788-8268-b27f2dc8f290-kube-api-access-twvcp\") pod \"payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7\" (UID: \"e9e8e764-309c-4788-8268-b27f2dc8f290\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:31.090933 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.090913    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e8e764-309c-4788-8268-b27f2dc8f290-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7\" (UID: \"e9e8e764-309c-4788-8268-b27f2dc8f290\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:31.109667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.109634    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"]
Nov 02 21:21:31.143768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.143692    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twvcp\" (UniqueName: \"kubernetes.io/projected/e9e8e764-309c-4788-8268-b27f2dc8f290-kube-api-access-twvcp\") pod \"payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7\" (UID: \"e9e8e764-309c-4788-8268-b27f2dc8f290\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:31.201150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.190176    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:31.201150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.192426    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc743a-7ada-48a6-95be-d6adfedbac23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4\" (UID: \"f7bc743a-7ada-48a6-95be-d6adfedbac23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:31.201150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.192463    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8h7\" (UniqueName: \"kubernetes.io/projected/f7bc743a-7ada-48a6-95be-d6adfedbac23-kube-api-access-rh8h7\") pod \"payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4\" (UID: \"f7bc743a-7ada-48a6-95be-d6adfedbac23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:31.294323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.293338    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc743a-7ada-48a6-95be-d6adfedbac23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4\" (UID: \"f7bc743a-7ada-48a6-95be-d6adfedbac23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:31.294323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.293389    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8h7\" (UniqueName: \"kubernetes.io/projected/f7bc743a-7ada-48a6-95be-d6adfedbac23-kube-api-access-rh8h7\") pod \"payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4\" (UID: \"f7bc743a-7ada-48a6-95be-d6adfedbac23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:31.294569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.294470    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc743a-7ada-48a6-95be-d6adfedbac23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4\" (UID: \"f7bc743a-7ada-48a6-95be-d6adfedbac23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:31.400180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.397638    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8h7\" (UniqueName: \"kubernetes.io/projected/f7bc743a-7ada-48a6-95be-d6adfedbac23-kube-api-access-rh8h7\") pod \"payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4\" (UID: \"f7bc743a-7ada-48a6-95be-d6adfedbac23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:31.578662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.578636    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"]
Nov 02 21:21:31.581594 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:21:31.581552    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e8e764_309c_4788_8268_b27f2dc8f290.slice/crio-7e636f4d436467d1865a2a6b92f90ac054bf18a2235229a7ddd923f7b805b33b WatchSource:0}: Error finding container 7e636f4d436467d1865a2a6b92f90ac054bf18a2235229a7ddd923f7b805b33b: Status 404 returned error can't find the container with id 7e636f4d436467d1865a2a6b92f90ac054bf18a2235229a7ddd923f7b805b33b
Nov 02 21:21:31.679827 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:31.679739    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:32.052354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:32.052168    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"]
Nov 02 21:21:32.055251 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:21:32.055225    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bc743a_7ada_48a6_95be_d6adfedbac23.slice/crio-c7b784839250f1517a3c550ebac3383e0914b18a4b2015c8afe6a47c369db493 WatchSource:0}: Error finding container c7b784839250f1517a3c550ebac3383e0914b18a4b2015c8afe6a47c369db493: Status 404 returned error can't find the container with id c7b784839250f1517a3c550ebac3383e0914b18a4b2015c8afe6a47c369db493
Nov 02 21:21:32.232144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:32.230501    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4" event={"ID":"f7bc743a-7ada-48a6-95be-d6adfedbac23","Type":"ContainerStarted","Data":"840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713"}
Nov 02 21:21:32.232144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:32.230579    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4" event={"ID":"f7bc743a-7ada-48a6-95be-d6adfedbac23","Type":"ContainerStarted","Data":"c7b784839250f1517a3c550ebac3383e0914b18a4b2015c8afe6a47c369db493"}
Nov 02 21:21:32.232144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:32.230752    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:32.232713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:32.232686    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7" event={"ID":"e9e8e764-309c-4788-8268-b27f2dc8f290","Type":"ContainerStarted","Data":"e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c"}
Nov 02 21:21:32.232823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:32.232729    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7" event={"ID":"e9e8e764-309c-4788-8268-b27f2dc8f290","Type":"ContainerStarted","Data":"7e636f4d436467d1865a2a6b92f90ac054bf18a2235229a7ddd923f7b805b33b"}
Nov 02 21:21:32.233214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:32.233197    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:32.336610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:32.336484    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4" podStartSLOduration=1.336465967 podStartE2EDuration="1.336465967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:32.335861369 +0000 UTC m=+779.599397686" watchObservedRunningTime="2026-11-02 21:21:32.336465967 +0000 UTC m=+779.600002283"
Nov 02 21:21:32.443406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:32.443346    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7" podStartSLOduration=2.443333026 podStartE2EDuration="2.443333026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:32.441772546 +0000 UTC m=+779.705308863" watchObservedRunningTime="2026-11-02 21:21:32.443333026 +0000 UTC m=+779.706869377"
Nov 02 21:21:39.337426 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.337380    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:21:39.723488 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.723415    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"]
Nov 02 21:21:39.731447 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.730984    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:39.782439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.782402    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"]
Nov 02 21:21:39.794195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.793921    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f052551-4d9f-455d-b26f-8f63f00a1885-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl\" (UID: \"2f052551-4d9f-455d-b26f-8f63f00a1885\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:39.794195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.794048    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lmld\" (UniqueName: \"kubernetes.io/projected/2f052551-4d9f-455d-b26f-8f63f00a1885-kube-api-access-6lmld\") pod \"payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl\" (UID: \"2f052551-4d9f-455d-b26f-8f63f00a1885\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:39.829094 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.828826    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5f765a\""
Nov 02 21:21:39.829094 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.829039    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5f765a-dockercfg-vkvgv\""
Nov 02 21:21:39.892712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.892678    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"]
Nov 02 21:21:39.894839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.894811    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lmld\" (UniqueName: \"kubernetes.io/projected/2f052551-4d9f-455d-b26f-8f63f00a1885-kube-api-access-6lmld\") pod \"payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl\" (UID: \"2f052551-4d9f-455d-b26f-8f63f00a1885\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:39.894976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.894947    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f052551-4d9f-455d-b26f-8f63f00a1885-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl\" (UID: \"2f052551-4d9f-455d-b26f-8f63f00a1885\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:39.895607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.895588    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f052551-4d9f-455d-b26f-8f63f00a1885-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl\" (UID: \"2f052551-4d9f-455d-b26f-8f63f00a1885\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:39.897342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.897324    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:39.947843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.947803    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"]
Nov 02 21:21:39.952273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.952242    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lmld\" (UniqueName: \"kubernetes.io/projected/2f052551-4d9f-455d-b26f-8f63f00a1885-kube-api-access-6lmld\") pod \"payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl\" (UID: \"2f052551-4d9f-455d-b26f-8f63f00a1885\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:39.996593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.996501    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092804d-6683-4b47-9d26-ad6d9d2df062-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx\" (UID: \"b092804d-6683-4b47-9d26-ad6d9d2df062\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:39.996593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:39.996548    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgzb4\" (UniqueName: \"kubernetes.io/projected/b092804d-6683-4b47-9d26-ad6d9d2df062-kube-api-access-mgzb4\") pod \"payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx\" (UID: \"b092804d-6683-4b47-9d26-ad6d9d2df062\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:40.044301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:40.044260    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:40.101296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:40.100704    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092804d-6683-4b47-9d26-ad6d9d2df062-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx\" (UID: \"b092804d-6683-4b47-9d26-ad6d9d2df062\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:40.101296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:40.100772    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgzb4\" (UniqueName: \"kubernetes.io/projected/b092804d-6683-4b47-9d26-ad6d9d2df062-kube-api-access-mgzb4\") pod \"payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx\" (UID: \"b092804d-6683-4b47-9d26-ad6d9d2df062\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:40.102062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:40.102031    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092804d-6683-4b47-9d26-ad6d9d2df062-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx\" (UID: \"b092804d-6683-4b47-9d26-ad6d9d2df062\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:40.158398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:40.158362    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgzb4\" (UniqueName: \"kubernetes.io/projected/b092804d-6683-4b47-9d26-ad6d9d2df062-kube-api-access-mgzb4\") pod \"payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx\" (UID: \"b092804d-6683-4b47-9d26-ad6d9d2df062\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:40.214340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:40.214306    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:40.419036 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:21:40.419006    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f052551_4d9f_455d_b26f_8f63f00a1885.slice/crio-908be547f0e44ecc7ba6ece73f10f35dab9c9bd609a638fa9a9b1a2a020b0808 WatchSource:0}: Error finding container 908be547f0e44ecc7ba6ece73f10f35dab9c9bd609a638fa9a9b1a2a020b0808: Status 404 returned error can't find the container with id 908be547f0e44ecc7ba6ece73f10f35dab9c9bd609a638fa9a9b1a2a020b0808
Nov 02 21:21:40.420697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:40.420674    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"]
Nov 02 21:21:40.585665 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:21:40.585632    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb092804d_6683_4b47_9d26_ad6d9d2df062.slice/crio-a055bc09fe22e156beb47ea55df23d29c6205f03a8519736171f86a25963259e WatchSource:0}: Error finding container a055bc09fe22e156beb47ea55df23d29c6205f03a8519736171f86a25963259e: Status 404 returned error can't find the container with id a055bc09fe22e156beb47ea55df23d29c6205f03a8519736171f86a25963259e
Nov 02 21:21:40.586009 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:40.585989    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"]
Nov 02 21:21:41.285378 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.285341    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl" event={"ID":"2f052551-4d9f-455d-b26f-8f63f00a1885","Type":"ContainerStarted","Data":"ceb5b58190e000694c62ea2ce887286c1c998602e8c9afba2537f276ac4a82c7"}
Nov 02 21:21:41.285606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.285407    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl" event={"ID":"2f052551-4d9f-455d-b26f-8f63f00a1885","Type":"ContainerStarted","Data":"908be547f0e44ecc7ba6ece73f10f35dab9c9bd609a638fa9a9b1a2a020b0808"}
Nov 02 21:21:41.285606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.285435    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:41.286920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.286888    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx" event={"ID":"b092804d-6683-4b47-9d26-ad6d9d2df062","Type":"ContainerStarted","Data":"b4b203b8a7f04f7185994362b19c739e1fafc6d1749aff9b6a732ecf21ede65a"}
Nov 02 21:21:41.287017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.286946    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:41.287017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.286972    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx" event={"ID":"b092804d-6683-4b47-9d26-ad6d9d2df062","Type":"ContainerStarted","Data":"a055bc09fe22e156beb47ea55df23d29c6205f03a8519736171f86a25963259e"}
Nov 02 21:21:41.288354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.288332    4528 generic.go:356] "Generic (PLEG): container finished" podID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerID="f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f" exitCode=0
Nov 02 21:21:41.288456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.288367    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" event={"ID":"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c","Type":"ContainerDied","Data":"f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f"}
Nov 02 21:21:41.288696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.288678    4528 scope.go:122] "RemoveContainer" containerID="f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f"
Nov 02 21:21:41.289923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.289899    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:41.289978 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.289964    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:41.290030 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.289995    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:21:41.400705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.400648    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl" podStartSLOduration=2.400630889 podStartE2EDuration="2.400630889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:41.399239663 +0000 UTC m=+788.662775993" watchObservedRunningTime="2026-11-02 21:21:41.400630889 +0000 UTC m=+788.664167207"
Nov 02 21:21:41.642925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:41.642820    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx" podStartSLOduration=2.6428049810000003 podStartE2EDuration="2.642804981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:41.522858853 +0000 UTC m=+788.786395172" watchObservedRunningTime="2026-11-02 21:21:41.642804981 +0000 UTC m=+788.906341348"
Nov 02 21:21:42.294755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:42.294721    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" event={"ID":"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c","Type":"ContainerStarted","Data":"fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6"}
Nov 02 21:21:42.972759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:42.972728    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:42.973209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:42.972942    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 21:21:42.973209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:42.973005    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 21:21:43.240357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:43.240273    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 21:21:43.972929 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:43.972886    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 21:21:43.973369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:43.972946    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 21:21:44.134815 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:21:44.134773    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:21:44.245042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:44.244936    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 21:21:44.972826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:44.972793    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 21:21:44.973029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:44.972842    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 21:21:45.973059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:45.973025    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 21:21:45.973561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:45.973079    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 21:21:46.972869 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:46.972835    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 21:21:46.973074 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:46.972885    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 21:21:47.976092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:47.976062    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:47.976491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:47.976353    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:47.976960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:47.976942    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:21:52.298395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:52.298357    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 21:21:52.298807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:52.298509    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 21:21:58.381253 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:21:58.381202    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:22:07.521437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:07.520127    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:44900->10.133.0.57:15021: read: connection reset by peer" start-of-body=
Nov 02 21:22:07.521437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:07.520200    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:44900->10.133.0.57:15021: read: connection reset by peer"
Nov 02 21:22:08.253326 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:08.253288    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:22:08.480072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:08.480035    4528 generic.go:356] "Generic (PLEG): container finished" podID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerID="b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf" exitCode=0
Nov 02 21:22:08.480355 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:08.480104    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" event={"ID":"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9","Type":"ContainerDied","Data":"b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf"}
Nov 02 21:22:08.480500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:08.480487    4528 scope.go:122] "RemoveContainer" containerID="b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf"
Nov 02 21:22:08.481716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:08.481684    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:22:08.481820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:08.481767    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:22:08.481882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:08.481815    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:22:09.489663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:09.489629    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" event={"ID":"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9","Type":"ContainerStarted","Data":"7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c"}
Nov 02 21:22:10.060742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:10.060707    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:22:10.060987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:10.060942    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 21:22:10.061072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:10.061031    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 21:22:11.060827 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:11.060789    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 21:22:11.061284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:11.060856    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 21:22:12.061456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:12.061420    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 21:22:12.061854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:12.061474    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 21:22:13.061166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:13.061104    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 21:22:13.061380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:13.061180    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 21:22:14.060892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:14.060858    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 21:22:14.061334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:14.060914    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 21:22:14.135975 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:22:14.135941    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:22:15.064572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:15.064542    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:22:15.064984 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:15.064799    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:22:15.065542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:15.065526    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:22:21.061760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:21.061729    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:22:35.106065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:35.106033    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:22:42.639564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.639532    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"]
Nov 02 21:22:42.643393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.643375    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.680016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.679975    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c8cb-561b-400d-83a7-80686d1ba923-tls-certs\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.680181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.680062    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-model-cache\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.680181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.680154    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-home\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.680286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.680211    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-dshm\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.680286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.680255    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bblg\" (UniqueName: \"kubernetes.io/projected/a5f1c8cb-561b-400d-83a7-80686d1ba923-kube-api-access-7bblg\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.680401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.680388    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-tmp-dir\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.706993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.706953    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"]
Nov 02 21:22:42.744691 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.744668    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"openshift-service-ca.crt\""
Nov 02 21:22:42.744691 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.744667    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"kube-root-ca.crt\""
Nov 02 21:22:42.746259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.746244    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"premium-w3-f51151-kserve-self-signed-certs\""
Nov 02 21:22:42.746317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.746307    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"default-dockercfg-2ss8g\""
Nov 02 21:22:42.781805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.781777    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c8cb-561b-400d-83a7-80686d1ba923-tls-certs\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.781937 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.781813    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-model-cache\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.781980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.781961    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-home\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.782020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.781994    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-dshm\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.782020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.782015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bblg\" (UniqueName: \"kubernetes.io/projected/a5f1c8cb-561b-400d-83a7-80686d1ba923-kube-api-access-7bblg\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.782093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.782071    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-tmp-dir\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.782247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.782230    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-model-cache\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.782371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.782346    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-home\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.782507 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.782491    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-tmp-dir\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.784443 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.784423    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c8cb-561b-400d-83a7-80686d1ba923-tls-certs\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.784536 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.784459    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-dshm\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.841281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.841248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bblg\" (UniqueName: \"kubernetes.io/projected/a5f1c8cb-561b-400d-83a7-80686d1ba923-kube-api-access-7bblg\") pod \"premium-w3-f51151-kserve-6cc686768-7vlmr\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") " pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:42.981324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:42.981241    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:43.348261 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:22:43.348227    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5f1c8cb_561b_400d_83a7_80686d1ba923.slice/crio-14026c8c5991742eecd2189f054110095f18536a07e9c6771bf45fa01fd8e9cb WatchSource:0}: Error finding container 14026c8c5991742eecd2189f054110095f18536a07e9c6771bf45fa01fd8e9cb: Status 404 returned error can't find the container with id 14026c8c5991742eecd2189f054110095f18536a07e9c6771bf45fa01fd8e9cb
Nov 02 21:22:43.350188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:43.350153    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"]
Nov 02 21:22:43.705554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:43.705472    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" event={"ID":"a5f1c8cb-561b-400d-83a7-80686d1ba923","Type":"ContainerStarted","Data":"6457b5cd8f24b204a945849050c0cb8d674fe12a7c24850107cf38ab7a132bf7"}
Nov 02 21:22:43.705554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:43.705522    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" event={"ID":"a5f1c8cb-561b-400d-83a7-80686d1ba923","Type":"ContainerStarted","Data":"14026c8c5991742eecd2189f054110095f18536a07e9c6771bf45fa01fd8e9cb"}
Nov 02 21:22:43.820462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:43.820402    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podStartSLOduration=1.820388426 podStartE2EDuration="1.820388426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:43.819664947 +0000 UTC m=+851.083201263" watchObservedRunningTime="2026-11-02 21:22:43.820388426 +0000 UTC m=+851.083924742"
Nov 02 21:22:43.981916 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:43.981834    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:44.112530 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:22:44.112498    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:22:48.647997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:48.647902    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:22:52.981901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:52.981863    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:52.994574 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:52.994550    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:22:53.771908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:22:53.771881    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:23:06.378766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:06.378729    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:23:14.114160 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:23:14.114088    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:23:17.071682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.071645    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"]
Nov 02 21:23:17.075504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.075486    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.152532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.152500    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tmp-dir\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.152532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.152536    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-home\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.152722 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.152665    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tls-certs\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.152758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.152719    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sc5p\" (UniqueName: \"kubernetes.io/projected/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-kube-api-access-5sc5p\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.152799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.152765    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-model-cache\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.152799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.152788    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-dshm\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.178640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.178599    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"default-dockercfg-rtxrz\""
Nov 02 21:23:17.178640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.178619    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"openshift-service-ca.crt\""
Nov 02 21:23:17.178640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.178599    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"kube-root-ca.crt\""
Nov 02 21:23:17.178878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.178665    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"premium-w4-d6cb38-kserve-self-signed-certs\""
Nov 02 21:23:17.180595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.180569    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"]
Nov 02 21:23:17.253620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.253588    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-model-cache\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.253620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.253626    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-dshm\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.253865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.253657    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tmp-dir\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.253865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.253680    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-home\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.253865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.253773    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tls-certs\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.253865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.253834    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sc5p\" (UniqueName: \"kubernetes.io/projected/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-kube-api-access-5sc5p\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.254203 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.254180    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tmp-dir\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.254309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.254201    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-model-cache\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.254309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.254251    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-home\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.256014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.255987    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-dshm\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.256191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.256175    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tls-certs\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.314269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.314233    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sc5p\" (UniqueName: \"kubernetes.io/projected/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-kube-api-access-5sc5p\") pod \"premium-w4-d6cb38-kserve-57466db89d-fssbt\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") " pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.387801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.387714    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:17.788674 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:23:17.788646    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57c65c87_4bcb_4a5c_b4d1_396f2a7f3612.slice/crio-eae3be9ff06920515ad56fc7e1c1f55dde01964a4979c29e63fb883a0fc8b572 WatchSource:0}: Error finding container eae3be9ff06920515ad56fc7e1c1f55dde01964a4979c29e63fb883a0fc8b572: Status 404 returned error can't find the container with id eae3be9ff06920515ad56fc7e1c1f55dde01964a4979c29e63fb883a0fc8b572
Nov 02 21:23:17.789850 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.789825    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"]
Nov 02 21:23:17.878775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.878743    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" event={"ID":"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612","Type":"ContainerStarted","Data":"a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe"}
Nov 02 21:23:17.878775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.878780    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" event={"ID":"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612","Type":"ContainerStarted","Data":"eae3be9ff06920515ad56fc7e1c1f55dde01964a4979c29e63fb883a0fc8b572"}
Nov 02 21:23:17.993763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:17.993706    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podStartSLOduration=0.993691503 podStartE2EDuration="993.691503ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:17.992504908 +0000 UTC m=+885.256041225" watchObservedRunningTime="2026-11-02 21:23:17.993691503 +0000 UTC m=+885.257227821"
Nov 02 21:23:18.387815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:18.387786    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:21.497441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:21.497407    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:23:27.388208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:27.388161    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:27.418792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:27.418762    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:27.937926 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:27.937898    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:23:43.615238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:43.615203    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:23:44.111990 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:23:44.111956    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:23:53.987524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:53.987490    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"]
Nov 02 21:23:53.991439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:53.991414    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.044063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.044027    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"]
Nov 02 21:23:54.092184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.092149    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"premium-w2-6fa533-kserve-self-signed-certs\""
Nov 02 21:23:54.092353 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.092204    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"default-dockercfg-lt567\""
Nov 02 21:23:54.093985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.093966    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"openshift-service-ca.crt\""
Nov 02 21:23:54.093985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.093966    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"kube-root-ca.crt\""
Nov 02 21:23:54.135256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.135224    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv7cb\" (UniqueName: \"kubernetes.io/projected/182772ef-d7fb-465f-94f6-20e4fc63e7f5-kube-api-access-sv7cb\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.135450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.135281    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tls-certs\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.135450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.135324    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-model-cache\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.135450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.135377    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-dshm\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.135450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.135433    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tmp-dir\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.135662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.135461    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-home\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.236275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.236244    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv7cb\" (UniqueName: \"kubernetes.io/projected/182772ef-d7fb-465f-94f6-20e4fc63e7f5-kube-api-access-sv7cb\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.236469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.236287    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tls-certs\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.236469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.236324    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-model-cache\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.236469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.236368    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-dshm\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.236469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.236416    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tmp-dir\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.236469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.236440    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-home\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.236911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.236884    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-model-cache\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.236911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.236906    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tmp-dir\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.237062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.236939    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-home\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.238841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.238779    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-dshm\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.239248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.239225    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tls-certs\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.293923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.293889    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv7cb\" (UniqueName: \"kubernetes.io/projected/182772ef-d7fb-465f-94f6-20e4fc63e7f5-kube-api-access-sv7cb\") pod \"premium-w2-6fa533-kserve-5c54778dd-6q2h6\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") " pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.303709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.303680    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:54.673452 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:23:54.673411    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod182772ef_d7fb_465f_94f6_20e4fc63e7f5.slice/crio-f9c4eb6956128cb6e897a067ff20e0c1d92d0c9e746316ebfb0a6548ce0d95bf WatchSource:0}: Error finding container f9c4eb6956128cb6e897a067ff20e0c1d92d0c9e746316ebfb0a6548ce0d95bf: Status 404 returned error can't find the container with id f9c4eb6956128cb6e897a067ff20e0c1d92d0c9e746316ebfb0a6548ce0d95bf
Nov 02 21:23:54.675378 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:54.675354    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"]
Nov 02 21:23:55.061182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:55.061139    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" event={"ID":"182772ef-d7fb-465f-94f6-20e4fc63e7f5","Type":"ContainerStarted","Data":"ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0"}
Nov 02 21:23:55.061585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:55.061196    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" event={"ID":"182772ef-d7fb-465f-94f6-20e4fc63e7f5","Type":"ContainerStarted","Data":"f9c4eb6956128cb6e897a067ff20e0c1d92d0c9e746316ebfb0a6548ce0d95bf"}
Nov 02 21:23:55.175726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:55.175677    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podStartSLOduration=2.17566301 podStartE2EDuration="2.17566301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:55.174025656 +0000 UTC m=+922.437561973" watchObservedRunningTime="2026-11-02 21:23:55.17566301 +0000 UTC m=+922.439199326"
Nov 02 21:23:55.304714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:55.304661    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:23:59.648322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.648287    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"]
Nov 02 21:23:59.652031 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.652013    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.696550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.696518    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-041927d0-ffttj\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.696550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.696556    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhh6l\" (UniqueName: \"kubernetes.io/projected/758b81bd-31a9-410e-868d-72ed8cdf9aa6-kube-api-access-qhh6l\") pod \"maas-api-revoke-sub-041927d0-ffttj\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.696758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.696576    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-041927d0-ffttj\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.704000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.703971    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"]
Nov 02 21:23:59.752362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.752336    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:23:59.752362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.752352    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:23:59.752573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.752336    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:23:59.752573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.752413    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:23:59.798268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.798233    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-041927d0-ffttj\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.798268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.798277    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhh6l\" (UniqueName: \"kubernetes.io/projected/758b81bd-31a9-410e-868d-72ed8cdf9aa6-kube-api-access-qhh6l\") pod \"maas-api-revoke-sub-041927d0-ffttj\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.798498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.798302    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-041927d0-ffttj\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.798939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.798915    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-041927d0-ffttj\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.800511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.800486    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-041927d0-ffttj\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.855341 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.855308    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhh6l\" (UniqueName: \"kubernetes.io/projected/758b81bd-31a9-410e-868d-72ed8cdf9aa6-kube-api-access-qhh6l\") pod \"maas-api-revoke-sub-041927d0-ffttj\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:23:59.965322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:23:59.965241    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:24:00.334310 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:00.334279    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"]
Nov 02 21:24:00.335349 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:24:00.335327    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758b81bd_31a9_410e_868d_72ed8cdf9aa6.slice/crio-3e189181747cb7a12bd415752310a74623442363e79797b64452630a021f74f8 WatchSource:0}: Error finding container 3e189181747cb7a12bd415752310a74623442363e79797b64452630a021f74f8: Status 404 returned error can't find the container with id 3e189181747cb7a12bd415752310a74623442363e79797b64452630a021f74f8
Nov 02 21:24:01.105617 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:01.105584    4528 generic.go:356] "Generic (PLEG): container finished" podID="758b81bd-31a9-410e-868d-72ed8cdf9aa6" containerID="1711e1288218a159bee4ae6ad1c255443f79cf2e3cddd1bc6a149141cbb0e03e" exitCode=0
Nov 02 21:24:01.105983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:01.105624    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj" event={"ID":"758b81bd-31a9-410e-868d-72ed8cdf9aa6","Type":"ContainerDied","Data":"1711e1288218a159bee4ae6ad1c255443f79cf2e3cddd1bc6a149141cbb0e03e"}
Nov 02 21:24:01.105983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:01.105663    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj" event={"ID":"758b81bd-31a9-410e-868d-72ed8cdf9aa6","Type":"ContainerStarted","Data":"3e189181747cb7a12bd415752310a74623442363e79797b64452630a021f74f8"}
Nov 02 21:24:02.251578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.251554    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:24:02.322362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.322326    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-openshift-ca\") pod \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") "
Nov 02 21:24:02.322553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.322417    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-xks-ca\") pod \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") "
Nov 02 21:24:02.322553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.322461    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758b81bd-31a9-410e-868d-72ed8cdf9aa6-kube-api-access-qhh6l\" (UniqueName: \"kubernetes.io/projected/758b81bd-31a9-410e-868d-72ed8cdf9aa6-kube-api-access-qhh6l\") pod \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\" (UID: \"758b81bd-31a9-410e-868d-72ed8cdf9aa6\") "
Nov 02 21:24:02.322700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.322676    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-openshift-ca" pod "758b81bd-31a9-410e-868d-72ed8cdf9aa6" (UID: "758b81bd-31a9-410e-868d-72ed8cdf9aa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:24:02.324516 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.324490    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758b81bd-31a9-410e-868d-72ed8cdf9aa6-kube-api-access-qhh6l" pod "758b81bd-31a9-410e-868d-72ed8cdf9aa6" (UID: "758b81bd-31a9-410e-868d-72ed8cdf9aa6"). InnerVolumeSpecName "kube-api-access-qhh6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:24:02.324624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.324517    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-xks-ca" pod "758b81bd-31a9-410e-868d-72ed8cdf9aa6" (UID: "758b81bd-31a9-410e-868d-72ed8cdf9aa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:24:02.423902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.423820    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhh6l\" (UniqueName: \"kubernetes.io/projected/758b81bd-31a9-410e-868d-72ed8cdf9aa6-kube-api-access-qhh6l\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:02.423902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.423852    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:02.423902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:02.423865    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758b81bd-31a9-410e-868d-72ed8cdf9aa6-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:03.130586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:03.130541    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj" event={"ID":"758b81bd-31a9-410e-868d-72ed8cdf9aa6","Type":"ContainerDied","Data":"3e189181747cb7a12bd415752310a74623442363e79797b64452630a021f74f8"}
Nov 02 21:24:03.130768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:03.130613    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e189181747cb7a12bd415752310a74623442363e79797b64452630a021f74f8"
Nov 02 21:24:03.130768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:03.130685    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"
Nov 02 21:24:04.303838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:04.303793    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:24:04.316418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:04.316386    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:24:05.151195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:05.151165    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:24:05.607132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:05.606559    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:24:14.113274 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:24:14.113238    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:24:17.416701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.416660    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"]
Nov 02 21:24:17.417205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.417193    4528 cpu_manager.go:414] "containerMap: removing container" podUID="758b81bd-31a9-410e-868d-72ed8cdf9aa6" containerName="revoke-keys"
Nov 02 21:24:17.417293 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.417207    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758b81bd-31a9-410e-868d-72ed8cdf9aa6" containerName="revoke-keys"
Nov 02 21:24:17.417359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.417308    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="758b81bd-31a9-410e-868d-72ed8cdf9aa6" containerName="revoke-keys"
Nov 02 21:24:17.420742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.420726    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.472698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.472663    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"]
Nov 02 21:24:17.494766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.494734    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb9c929-thsvk\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.494944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.494776    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb9c929-thsvk\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.494944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.494797    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrqft\" (UniqueName: \"kubernetes.io/projected/fc86f8b1-623c-4bad-853f-39b58ee0af44-kube-api-access-xrqft\") pod \"maas-api-revoke-sub-bfb9c929-thsvk\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.508444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.508416    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:24:17.520810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.520788    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:24:17.520810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.520788    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:24:17.520951 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.520841    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:24:17.596180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.596143    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb9c929-thsvk\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.596388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.596207    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb9c929-thsvk\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.596388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.596233    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrqft\" (UniqueName: \"kubernetes.io/projected/fc86f8b1-623c-4bad-853f-39b58ee0af44-kube-api-access-xrqft\") pod \"maas-api-revoke-sub-bfb9c929-thsvk\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.596898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.596869    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb9c929-thsvk\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.598529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.598507    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb9c929-thsvk\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.653996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.653958    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrqft\" (UniqueName: \"kubernetes.io/projected/fc86f8b1-623c-4bad-853f-39b58ee0af44-kube-api-access-xrqft\") pod \"maas-api-revoke-sub-bfb9c929-thsvk\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:17.733668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:17.733586    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:18.099463 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:24:18.099431    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc86f8b1_623c_4bad_853f_39b58ee0af44.slice/crio-7da097dc8702c05e13102e4e2c17473fd95ee630e8328a99d763bf4f81d1ca8a WatchSource:0}: Error finding container 7da097dc8702c05e13102e4e2c17473fd95ee630e8328a99d763bf4f81d1ca8a: Status 404 returned error can't find the container with id 7da097dc8702c05e13102e4e2c17473fd95ee630e8328a99d763bf4f81d1ca8a
Nov 02 21:24:18.101290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:18.101265    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"]
Nov 02 21:24:18.228013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:18.227978    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk" event={"ID":"fc86f8b1-623c-4bad-853f-39b58ee0af44","Type":"ContainerStarted","Data":"7da097dc8702c05e13102e4e2c17473fd95ee630e8328a99d763bf4f81d1ca8a"}
Nov 02 21:24:19.237266 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:19.237178    4528 generic.go:356] "Generic (PLEG): container finished" podID="fc86f8b1-623c-4bad-853f-39b58ee0af44" containerID="9aca9a93cab1c2c0537badca3f80b66b814add25cc181802185c1ff3b1acc3a0" exitCode=0
Nov 02 21:24:19.237630 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:19.237270    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk" event={"ID":"fc86f8b1-623c-4bad-853f-39b58ee0af44","Type":"ContainerDied","Data":"9aca9a93cab1c2c0537badca3f80b66b814add25cc181802185c1ff3b1acc3a0"}
Nov 02 21:24:20.401707 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.401680    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:20.538469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.538431    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-xks-ca\") pod \"fc86f8b1-623c-4bad-853f-39b58ee0af44\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") "
Nov 02 21:24:20.538654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.538491    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc86f8b1-623c-4bad-853f-39b58ee0af44-kube-api-access-xrqft\" (UniqueName: \"kubernetes.io/projected/fc86f8b1-623c-4bad-853f-39b58ee0af44-kube-api-access-xrqft\") pod \"fc86f8b1-623c-4bad-853f-39b58ee0af44\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") "
Nov 02 21:24:20.538654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.538528    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-openshift-ca\") pod \"fc86f8b1-623c-4bad-853f-39b58ee0af44\" (UID: \"fc86f8b1-623c-4bad-853f-39b58ee0af44\") "
Nov 02 21:24:20.539208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.539175    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-openshift-ca" pod "fc86f8b1-623c-4bad-853f-39b58ee0af44" (UID: "fc86f8b1-623c-4bad-853f-39b58ee0af44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:24:20.540889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.540864    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-xks-ca" pod "fc86f8b1-623c-4bad-853f-39b58ee0af44" (UID: "fc86f8b1-623c-4bad-853f-39b58ee0af44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:24:20.544669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.544640    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc86f8b1-623c-4bad-853f-39b58ee0af44-kube-api-access-xrqft" pod "fc86f8b1-623c-4bad-853f-39b58ee0af44" (UID: "fc86f8b1-623c-4bad-853f-39b58ee0af44"). InnerVolumeSpecName "kube-api-access-xrqft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:24:20.639606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.639560    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:20.639606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.639602    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrqft\" (UniqueName: \"kubernetes.io/projected/fc86f8b1-623c-4bad-853f-39b58ee0af44-kube-api-access-xrqft\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:20.639606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:20.639613    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc86f8b1-623c-4bad-853f-39b58ee0af44-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:21.249935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:21.249850    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"
Nov 02 21:24:21.249935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:21.249860    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk" event={"ID":"fc86f8b1-623c-4bad-853f-39b58ee0af44","Type":"ContainerDied","Data":"7da097dc8702c05e13102e4e2c17473fd95ee630e8328a99d763bf4f81d1ca8a"}
Nov 02 21:24:21.249935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:21.249916    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7da097dc8702c05e13102e4e2c17473fd95ee630e8328a99d763bf4f81d1ca8a"
Nov 02 21:24:34.675794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.675761    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"]
Nov 02 21:24:34.676217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.676195    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fc86f8b1-623c-4bad-853f-39b58ee0af44" containerName="revoke-keys"
Nov 02 21:24:34.676217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.676207    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc86f8b1-623c-4bad-853f-39b58ee0af44" containerName="revoke-keys"
Nov 02 21:24:34.676299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.676266    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc86f8b1-623c-4bad-853f-39b58ee0af44" containerName="revoke-keys"
Nov 02 21:24:34.679758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.679736    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.769494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.769460    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdvp7\" (UniqueName: \"kubernetes.io/projected/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-kube-api-access-fdvp7\") pod \"maas-api-revoke-sub-d2e1b58c-zcj8w\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.769683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.769591    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e1b58c-zcj8w\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.769683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.769627    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e1b58c-zcj8w\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.783022 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.782987    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:24:34.783230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.783055    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:24:34.783230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.783164    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:24:34.783230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.783168    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:24:34.785167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.785146    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"]
Nov 02 21:24:34.870606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.870572    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e1b58c-zcj8w\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.870779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.870613    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e1b58c-zcj8w\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.870779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.870687    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdvp7\" (UniqueName: \"kubernetes.io/projected/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-kube-api-access-fdvp7\") pod \"maas-api-revoke-sub-d2e1b58c-zcj8w\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.871379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.871356    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e1b58c-zcj8w\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.872972 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.872933    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e1b58c-zcj8w\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.901694 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.901661    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:24:34.927759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.927694    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdvp7\" (UniqueName: \"kubernetes.io/projected/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-kube-api-access-fdvp7\") pod \"maas-api-revoke-sub-d2e1b58c-zcj8w\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:34.992096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:34.992065    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:35.373206 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:24:35.373177    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ceaee2_7bc0_4fcc_b83c_75595a5548be.slice/crio-15d51503f32e854968dc47da287e35093b9e13ac5c1c43cb3224993b6f4f6ef0 WatchSource:0}: Error finding container 15d51503f32e854968dc47da287e35093b9e13ac5c1c43cb3224993b6f4f6ef0: Status 404 returned error can't find the container with id 15d51503f32e854968dc47da287e35093b9e13ac5c1c43cb3224993b6f4f6ef0
Nov 02 21:24:35.374373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.374344    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"]
Nov 02 21:24:35.394357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.394320    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"]
Nov 02 21:24:35.401634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.401606    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.446159 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.446124    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"]
Nov 02 21:24:35.476995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.476939    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a5c342-ws9ml\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.476995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.476996    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a5c342-ws9ml\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.477216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.477077    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9m8b\" (UniqueName: \"kubernetes.io/projected/ff28f90a-5574-4eb2-bfff-bc905d6be61f-kube-api-access-n9m8b\") pod \"maas-api-revoke-sub-b4a5c342-ws9ml\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.578046 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.578015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a5c342-ws9ml\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.578046 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.578050    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a5c342-ws9ml\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.578312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.578071    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9m8b\" (UniqueName: \"kubernetes.io/projected/ff28f90a-5574-4eb2-bfff-bc905d6be61f-kube-api-access-n9m8b\") pod \"maas-api-revoke-sub-b4a5c342-ws9ml\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.578803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.578780    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a5c342-ws9ml\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.580434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.580413    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a5c342-ws9ml\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.635251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.635181    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9m8b\" (UniqueName: \"kubernetes.io/projected/ff28f90a-5574-4eb2-bfff-bc905d6be61f-kube-api-access-n9m8b\") pod \"maas-api-revoke-sub-b4a5c342-ws9ml\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:35.722337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:35.722311    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:36.089095 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:24:36.089055    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff28f90a_5574_4eb2_bfff_bc905d6be61f.slice/crio-acf1bc81a8150d69f600415eb84a4910f705d461328829e10b157ea1ea8099b1 WatchSource:0}: Error finding container acf1bc81a8150d69f600415eb84a4910f705d461328829e10b157ea1ea8099b1: Status 404 returned error can't find the container with id acf1bc81a8150d69f600415eb84a4910f705d461328829e10b157ea1ea8099b1
Nov 02 21:24:36.091259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:36.091238    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"]
Nov 02 21:24:36.332651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:36.332621    4528 generic.go:356] "Generic (PLEG): container finished" podID="c8ceaee2-7bc0-4fcc-b83c-75595a5548be" containerID="dc54ac4779fd76ee780253a8c88d9a006dce8d0aa52d091f49102cdad63ba105" exitCode=0
Nov 02 21:24:36.332833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:36.332703    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w" event={"ID":"c8ceaee2-7bc0-4fcc-b83c-75595a5548be","Type":"ContainerDied","Data":"dc54ac4779fd76ee780253a8c88d9a006dce8d0aa52d091f49102cdad63ba105"}
Nov 02 21:24:36.332833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:36.332756    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w" event={"ID":"c8ceaee2-7bc0-4fcc-b83c-75595a5548be","Type":"ContainerStarted","Data":"15d51503f32e854968dc47da287e35093b9e13ac5c1c43cb3224993b6f4f6ef0"}
Nov 02 21:24:36.333995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:36.333972    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml" event={"ID":"ff28f90a-5574-4eb2-bfff-bc905d6be61f","Type":"ContainerStarted","Data":"acf1bc81a8150d69f600415eb84a4910f705d461328829e10b157ea1ea8099b1"}
Nov 02 21:24:37.340513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.340482    4528 generic.go:356] "Generic (PLEG): container finished" podID="ff28f90a-5574-4eb2-bfff-bc905d6be61f" containerID="4cc8a09a312ee5192188bc8c468a39e97821ef643b718e0b9e4eefbee51267af" exitCode=0
Nov 02 21:24:37.340984 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.340571    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml" event={"ID":"ff28f90a-5574-4eb2-bfff-bc905d6be61f","Type":"ContainerDied","Data":"4cc8a09a312ee5192188bc8c468a39e97821ef643b718e0b9e4eefbee51267af"}
Nov 02 21:24:37.480096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.480069    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:37.495331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.495300    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-openshift-ca\") pod \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") "
Nov 02 21:24:37.495446 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.495350    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-xks-ca\") pod \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") "
Nov 02 21:24:37.495446 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.495423    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-kube-api-access-fdvp7\" (UniqueName: \"kubernetes.io/projected/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-kube-api-access-fdvp7\") pod \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\" (UID: \"c8ceaee2-7bc0-4fcc-b83c-75595a5548be\") "
Nov 02 21:24:37.495651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.495626    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-openshift-ca" pod "c8ceaee2-7bc0-4fcc-b83c-75595a5548be" (UID: "c8ceaee2-7bc0-4fcc-b83c-75595a5548be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:24:37.495834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.495810    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:37.497481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.497456    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-xks-ca" pod "c8ceaee2-7bc0-4fcc-b83c-75595a5548be" (UID: "c8ceaee2-7bc0-4fcc-b83c-75595a5548be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:24:37.497616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.497594    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-kube-api-access-fdvp7" pod "c8ceaee2-7bc0-4fcc-b83c-75595a5548be" (UID: "c8ceaee2-7bc0-4fcc-b83c-75595a5548be"). InnerVolumeSpecName "kube-api-access-fdvp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:24:37.596569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.596490    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:37.596569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:37.596525    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdvp7\" (UniqueName: \"kubernetes.io/projected/c8ceaee2-7bc0-4fcc-b83c-75595a5548be-kube-api-access-fdvp7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:38.352515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.352478    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"
Nov 02 21:24:38.353982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.352930    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w" event={"ID":"c8ceaee2-7bc0-4fcc-b83c-75595a5548be","Type":"ContainerDied","Data":"15d51503f32e854968dc47da287e35093b9e13ac5c1c43cb3224993b6f4f6ef0"}
Nov 02 21:24:38.353982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.353018    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15d51503f32e854968dc47da287e35093b9e13ac5c1c43cb3224993b6f4f6ef0"
Nov 02 21:24:38.493907 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.493884    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:38.604768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.604660    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-xks-ca\") pod \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") "
Nov 02 21:24:38.604962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.604778    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff28f90a-5574-4eb2-bfff-bc905d6be61f-kube-api-access-n9m8b\" (UniqueName: \"kubernetes.io/projected/ff28f90a-5574-4eb2-bfff-bc905d6be61f-kube-api-access-n9m8b\") pod \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") "
Nov 02 21:24:38.604962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.604832    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-openshift-ca\") pod \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\" (UID: \"ff28f90a-5574-4eb2-bfff-bc905d6be61f\") "
Nov 02 21:24:38.605429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.605397    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-openshift-ca" pod "ff28f90a-5574-4eb2-bfff-bc905d6be61f" (UID: "ff28f90a-5574-4eb2-bfff-bc905d6be61f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:24:38.607509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.607474    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-xks-ca" pod "ff28f90a-5574-4eb2-bfff-bc905d6be61f" (UID: "ff28f90a-5574-4eb2-bfff-bc905d6be61f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:24:38.607509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.607499    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff28f90a-5574-4eb2-bfff-bc905d6be61f-kube-api-access-n9m8b" pod "ff28f90a-5574-4eb2-bfff-bc905d6be61f" (UID: "ff28f90a-5574-4eb2-bfff-bc905d6be61f"). InnerVolumeSpecName "kube-api-access-n9m8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:24:38.706552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.706511    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:38.706552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.706545    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9m8b\" (UniqueName: \"kubernetes.io/projected/ff28f90a-5574-4eb2-bfff-bc905d6be61f-kube-api-access-n9m8b\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:38.706552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:38.706555    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff28f90a-5574-4eb2-bfff-bc905d6be61f-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:39.359030 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:39.358996    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"
Nov 02 21:24:39.359030 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:39.359021    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml" event={"ID":"ff28f90a-5574-4eb2-bfff-bc905d6be61f","Type":"ContainerDied","Data":"acf1bc81a8150d69f600415eb84a4910f705d461328829e10b157ea1ea8099b1"}
Nov 02 21:24:39.359483 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:39.359094    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acf1bc81a8150d69f600415eb84a4910f705d461328829e10b157ea1ea8099b1"
Nov 02 21:24:40.282129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.282086    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"]
Nov 02 21:24:40.282612 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.282594    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ff28f90a-5574-4eb2-bfff-bc905d6be61f" containerName="revoke-keys"
Nov 02 21:24:40.282705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.282613    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff28f90a-5574-4eb2-bfff-bc905d6be61f" containerName="revoke-keys"
Nov 02 21:24:40.282705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.282627    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c8ceaee2-7bc0-4fcc-b83c-75595a5548be" containerName="revoke-keys"
Nov 02 21:24:40.282705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.282634    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ceaee2-7bc0-4fcc-b83c-75595a5548be" containerName="revoke-keys"
Nov 02 21:24:40.282856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.282740    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff28f90a-5574-4eb2-bfff-bc905d6be61f" containerName="revoke-keys"
Nov 02 21:24:40.282856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.282753    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ceaee2-7bc0-4fcc-b83c-75595a5548be" containerName="revoke-keys"
Nov 02 21:24:40.285875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.285854    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.322224 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.322198    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmr6\" (UniqueName: \"kubernetes.io/projected/89c7b2bc-0c36-46db-982b-b6692dd039db-kube-api-access-jnmr6\") pod \"maas-api-revoke-sub-604c3c35-q6t8c\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.322403 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.322238    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-604c3c35-q6t8c\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.322448 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.322393    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-604c3c35-q6t8c\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.338791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.338768    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:24:40.338939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.338890    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:24:40.388930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.388898    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:24:40.389438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.389168    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:24:40.391389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.391363    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"]
Nov 02 21:24:40.422899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.422867    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-604c3c35-q6t8c\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.423062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.422937    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmr6\" (UniqueName: \"kubernetes.io/projected/89c7b2bc-0c36-46db-982b-b6692dd039db-kube-api-access-jnmr6\") pod \"maas-api-revoke-sub-604c3c35-q6t8c\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.423062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.422969    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-604c3c35-q6t8c\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.423542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.423521    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-604c3c35-q6t8c\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.425252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.425230    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-604c3c35-q6t8c\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.479711 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.479670    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmr6\" (UniqueName: \"kubernetes.io/projected/89c7b2bc-0c36-46db-982b-b6692dd039db-kube-api-access-jnmr6\") pod \"maas-api-revoke-sub-604c3c35-q6t8c\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.600016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.599925    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:40.967292 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:24:40.967254    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c7b2bc_0c36_46db_982b_b6692dd039db.slice/crio-a9e94763b628e4a54b5265b74bb2abbdb301dcf4aab70d5de13b937cc6f20f98 WatchSource:0}: Error finding container a9e94763b628e4a54b5265b74bb2abbdb301dcf4aab70d5de13b937cc6f20f98: Status 404 returned error can't find the container with id a9e94763b628e4a54b5265b74bb2abbdb301dcf4aab70d5de13b937cc6f20f98
Nov 02 21:24:40.969205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:40.969181    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"]
Nov 02 21:24:41.372579 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:41.372544    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c" event={"ID":"89c7b2bc-0c36-46db-982b-b6692dd039db","Type":"ContainerStarted","Data":"cbfd66d311c2bdffb24763f02165e094ecb51b6db64656d480f99a553c5bdedf"}
Nov 02 21:24:41.372746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:41.372595    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c" event={"ID":"89c7b2bc-0c36-46db-982b-b6692dd039db","Type":"ContainerStarted","Data":"a9e94763b628e4a54b5265b74bb2abbdb301dcf4aab70d5de13b937cc6f20f98"}
Nov 02 21:24:41.489743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:41.489685    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c" podStartSLOduration=1.489670931 podStartE2EDuration="1.489670931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:41.488266718 +0000 UTC m=+968.751803035" watchObservedRunningTime="2026-11-02 21:24:41.489670931 +0000 UTC m=+968.753207248"
Nov 02 21:24:42.378723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:42.378687    4528 generic.go:356] "Generic (PLEG): container finished" podID="89c7b2bc-0c36-46db-982b-b6692dd039db" containerID="cbfd66d311c2bdffb24763f02165e094ecb51b6db64656d480f99a553c5bdedf" exitCode=0
Nov 02 21:24:42.378918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:42.378772    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c" event={"ID":"89c7b2bc-0c36-46db-982b-b6692dd039db","Type":"ContainerDied","Data":"cbfd66d311c2bdffb24763f02165e094ecb51b6db64656d480f99a553c5bdedf"}
Nov 02 21:24:43.519204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.519182    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:43.555232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.555203    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-openshift-ca\") pod \"89c7b2bc-0c36-46db-982b-b6692dd039db\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") "
Nov 02 21:24:43.555369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.555257    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-xks-ca\") pod \"89c7b2bc-0c36-46db-982b-b6692dd039db\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") "
Nov 02 21:24:43.555414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.555371    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c7b2bc-0c36-46db-982b-b6692dd039db-kube-api-access-jnmr6\" (UniqueName: \"kubernetes.io/projected/89c7b2bc-0c36-46db-982b-b6692dd039db-kube-api-access-jnmr6\") pod \"89c7b2bc-0c36-46db-982b-b6692dd039db\" (UID: \"89c7b2bc-0c36-46db-982b-b6692dd039db\") "
Nov 02 21:24:43.555592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.555558    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-openshift-ca" pod "89c7b2bc-0c36-46db-982b-b6692dd039db" (UID: "89c7b2bc-0c36-46db-982b-b6692dd039db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:24:43.555797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.555778    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:43.557221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.557197    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-xks-ca" pod "89c7b2bc-0c36-46db-982b-b6692dd039db" (UID: "89c7b2bc-0c36-46db-982b-b6692dd039db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:24:43.557313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.557296    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c7b2bc-0c36-46db-982b-b6692dd039db-kube-api-access-jnmr6" pod "89c7b2bc-0c36-46db-982b-b6692dd039db" (UID: "89c7b2bc-0c36-46db-982b-b6692dd039db"). InnerVolumeSpecName "kube-api-access-jnmr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:24:43.656525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.656453    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7b2bc-0c36-46db-982b-b6692dd039db-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:43.656525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:43.656480    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmr6\" (UniqueName: \"kubernetes.io/projected/89c7b2bc-0c36-46db-982b-b6692dd039db-kube-api-access-jnmr6\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:44.113555 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:24:44.113522    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:24:44.390860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:44.390781    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"
Nov 02 21:24:44.391010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:44.390786    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c" event={"ID":"89c7b2bc-0c36-46db-982b-b6692dd039db","Type":"ContainerDied","Data":"a9e94763b628e4a54b5265b74bb2abbdb301dcf4aab70d5de13b937cc6f20f98"}
Nov 02 21:24:44.391010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:44.390905    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9e94763b628e4a54b5265b74bb2abbdb301dcf4aab70d5de13b937cc6f20f98"
Nov 02 21:24:50.221170 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.221137    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"]
Nov 02 21:24:50.221603 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.221589    4528 cpu_manager.go:414] "containerMap: removing container" podUID="89c7b2bc-0c36-46db-982b-b6692dd039db" containerName="revoke-keys"
Nov 02 21:24:50.221652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.221604    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c7b2bc-0c36-46db-982b-b6692dd039db" containerName="revoke-keys"
Nov 02 21:24:50.221694 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.221685    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c7b2bc-0c36-46db-982b-b6692dd039db" containerName="revoke-keys"
Nov 02 21:24:50.224697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.224677    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.277763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.277726    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"]
Nov 02 21:24:50.324517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.324471    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:24:50.324517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.324471    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:24:50.324517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.324508    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:24:50.324801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.324784    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kddh\" (UniqueName: \"kubernetes.io/projected/feacee19-1354-434e-a316-f4f1bd8e41f6-kube-api-access-5kddh\") pod \"maas-api-revoke-sub-69c03c89-wh678\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.324848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.324829    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69c03c89-wh678\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.324969 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.324931    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69c03c89-wh678\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.327777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.327764    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:24:50.426132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.426071    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kddh\" (UniqueName: \"kubernetes.io/projected/feacee19-1354-434e-a316-f4f1bd8e41f6-kube-api-access-5kddh\") pod \"maas-api-revoke-sub-69c03c89-wh678\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.426319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.426167    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69c03c89-wh678\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.426319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.426241    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69c03c89-wh678\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.427000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.426973    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69c03c89-wh678\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.428535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.428512    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69c03c89-wh678\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.483449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.483381    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kddh\" (UniqueName: \"kubernetes.io/projected/feacee19-1354-434e-a316-f4f1bd8e41f6-kube-api-access-5kddh\") pod \"maas-api-revoke-sub-69c03c89-wh678\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.538679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.538648    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:50.919586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:50.919546    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"]
Nov 02 21:24:50.925234 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:24:50.925208    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeacee19_1354_434e_a316_f4f1bd8e41f6.slice/crio-7b70c7199df2c56e0724bae72ed67d2c353d3e1771ccc50338260927533e04b8 WatchSource:0}: Error finding container 7b70c7199df2c56e0724bae72ed67d2c353d3e1771ccc50338260927533e04b8: Status 404 returned error can't find the container with id 7b70c7199df2c56e0724bae72ed67d2c353d3e1771ccc50338260927533e04b8
Nov 02 21:24:51.080583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.080547    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"]
Nov 02 21:24:51.086572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.086542    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.136704 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.136666    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"]
Nov 02 21:24:51.139606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.137366    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6pv\" (UniqueName: \"kubernetes.io/projected/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-kube-api-access-st6pv\") pod \"maas-api-revoke-sub-b6eb4183-q8b2n\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.139606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.137495    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6eb4183-q8b2n\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.139606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.137550    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6eb4183-q8b2n\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.242289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.238695    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6pv\" (UniqueName: \"kubernetes.io/projected/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-kube-api-access-st6pv\") pod \"maas-api-revoke-sub-b6eb4183-q8b2n\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.242289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.238784    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6eb4183-q8b2n\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.242289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.238821    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6eb4183-q8b2n\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.242289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.239670    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6eb4183-q8b2n\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.243127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.242400    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6eb4183-q8b2n\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.295695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.295661    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6pv\" (UniqueName: \"kubernetes.io/projected/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-kube-api-access-st6pv\") pod \"maas-api-revoke-sub-b6eb4183-q8b2n\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.400440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.400407    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:51.451023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.450389    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678" event={"ID":"feacee19-1354-434e-a316-f4f1bd8e41f6","Type":"ContainerStarted","Data":"539acbb82e10e006695530023eda6795974116193f9bd7995fbc73e46d1923f2"}
Nov 02 21:24:51.451023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.450461    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678" event={"ID":"feacee19-1354-434e-a316-f4f1bd8e41f6","Type":"ContainerStarted","Data":"7b70c7199df2c56e0724bae72ed67d2c353d3e1771ccc50338260927533e04b8"}
Nov 02 21:24:51.797842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.797694    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678" podStartSLOduration=1.7976723510000001 podStartE2EDuration="1.797672351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:51.598407923 +0000 UTC m=+978.861944241" watchObservedRunningTime="2026-11-02 21:24:51.797672351 +0000 UTC m=+979.061208670"
Nov 02 21:24:51.821689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:51.821655    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"]
Nov 02 21:24:51.822596 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:24:51.822567    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a1c4c9_a5ca_4992_b21b_41c774df7df4.slice/crio-f43c4242bd90edce5531262cd98538551a8d1808cdcc7bde1be17d3de11a5f00 WatchSource:0}: Error finding container f43c4242bd90edce5531262cd98538551a8d1808cdcc7bde1be17d3de11a5f00: Status 404 returned error can't find the container with id f43c4242bd90edce5531262cd98538551a8d1808cdcc7bde1be17d3de11a5f00
Nov 02 21:24:52.457895 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:52.457801    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n" event={"ID":"f6a1c4c9-a5ca-4992-b21b-41c774df7df4","Type":"ContainerStarted","Data":"c27f32c7a7803338690e057fd5f5da891d702cff8f9a1c4dc17d59ff57efe6bd"}
Nov 02 21:24:52.457895 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:52.457862    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n" event={"ID":"f6a1c4c9-a5ca-4992-b21b-41c774df7df4","Type":"ContainerStarted","Data":"f43c4242bd90edce5531262cd98538551a8d1808cdcc7bde1be17d3de11a5f00"}
Nov 02 21:24:52.459430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:52.459404    4528 generic.go:356] "Generic (PLEG): container finished" podID="feacee19-1354-434e-a316-f4f1bd8e41f6" containerID="539acbb82e10e006695530023eda6795974116193f9bd7995fbc73e46d1923f2" exitCode=0
Nov 02 21:24:52.459548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:52.459481    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678" event={"ID":"feacee19-1354-434e-a316-f4f1bd8e41f6","Type":"ContainerDied","Data":"539acbb82e10e006695530023eda6795974116193f9bd7995fbc73e46d1923f2"}
Nov 02 21:24:52.572361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:52.572311    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n" podStartSLOduration=1.572296999 podStartE2EDuration="1.572296999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:52.570952134 +0000 UTC m=+979.834488452" watchObservedRunningTime="2026-11-02 21:24:52.572296999 +0000 UTC m=+979.835833316"
Nov 02 21:24:53.466782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.466740    4528 generic.go:356] "Generic (PLEG): container finished" podID="f6a1c4c9-a5ca-4992-b21b-41c774df7df4" containerID="c27f32c7a7803338690e057fd5f5da891d702cff8f9a1c4dc17d59ff57efe6bd" exitCode=0
Nov 02 21:24:53.467224 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.466837    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n" event={"ID":"f6a1c4c9-a5ca-4992-b21b-41c774df7df4","Type":"ContainerDied","Data":"c27f32c7a7803338690e057fd5f5da891d702cff8f9a1c4dc17d59ff57efe6bd"}
Nov 02 21:24:53.638884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.638850    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:53.688292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.688252    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-openshift-ca\") pod \"feacee19-1354-434e-a316-f4f1bd8e41f6\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") "
Nov 02 21:24:53.688292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.688291    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-xks-ca\") pod \"feacee19-1354-434e-a316-f4f1bd8e41f6\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") "
Nov 02 21:24:53.688540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.688408    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feacee19-1354-434e-a316-f4f1bd8e41f6-kube-api-access-5kddh\" (UniqueName: \"kubernetes.io/projected/feacee19-1354-434e-a316-f4f1bd8e41f6-kube-api-access-5kddh\") pod \"feacee19-1354-434e-a316-f4f1bd8e41f6\" (UID: \"feacee19-1354-434e-a316-f4f1bd8e41f6\") "
Nov 02 21:24:53.688674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.688647    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-openshift-ca" pod "feacee19-1354-434e-a316-f4f1bd8e41f6" (UID: "feacee19-1354-434e-a316-f4f1bd8e41f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:24:53.688837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.688820    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:53.690545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.690520    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-xks-ca" pod "feacee19-1354-434e-a316-f4f1bd8e41f6" (UID: "feacee19-1354-434e-a316-f4f1bd8e41f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:24:53.690663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.690542    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feacee19-1354-434e-a316-f4f1bd8e41f6-kube-api-access-5kddh" pod "feacee19-1354-434e-a316-f4f1bd8e41f6" (UID: "feacee19-1354-434e-a316-f4f1bd8e41f6"). InnerVolumeSpecName "kube-api-access-5kddh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:24:53.789668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.789628    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kddh\" (UniqueName: \"kubernetes.io/projected/feacee19-1354-434e-a316-f4f1bd8e41f6-kube-api-access-5kddh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:53.789668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:53.789658    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feacee19-1354-434e-a316-f4f1bd8e41f6-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:54.474977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.474952    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"
Nov 02 21:24:54.475371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.474987    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678" event={"ID":"feacee19-1354-434e-a316-f4f1bd8e41f6","Type":"ContainerDied","Data":"7b70c7199df2c56e0724bae72ed67d2c353d3e1771ccc50338260927533e04b8"}
Nov 02 21:24:54.475371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.475030    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b70c7199df2c56e0724bae72ed67d2c353d3e1771ccc50338260927533e04b8"
Nov 02 21:24:54.624805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.624781    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:54.700250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.700218    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-openshift-ca\") pod \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") "
Nov 02 21:24:54.700401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.700295    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-kube-api-access-st6pv\" (UniqueName: \"kubernetes.io/projected/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-kube-api-access-st6pv\") pod \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") "
Nov 02 21:24:54.700401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.700346    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-xks-ca\") pod \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\" (UID: \"f6a1c4c9-a5ca-4992-b21b-41c774df7df4\") "
Nov 02 21:24:54.700601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.700579    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-openshift-ca" pod "f6a1c4c9-a5ca-4992-b21b-41c774df7df4" (UID: "f6a1c4c9-a5ca-4992-b21b-41c774df7df4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:24:54.700702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.700687    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:54.702456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.702432    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-xks-ca" pod "f6a1c4c9-a5ca-4992-b21b-41c774df7df4" (UID: "f6a1c4c9-a5ca-4992-b21b-41c774df7df4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:24:54.702570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.702549    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-kube-api-access-st6pv" pod "f6a1c4c9-a5ca-4992-b21b-41c774df7df4" (UID: "f6a1c4c9-a5ca-4992-b21b-41c774df7df4"). InnerVolumeSpecName "kube-api-access-st6pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:24:54.801900 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.801867    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:54.801900 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:54.801896    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st6pv\" (UniqueName: \"kubernetes.io/projected/f6a1c4c9-a5ca-4992-b21b-41c774df7df4-kube-api-access-st6pv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:24:55.481643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:55.481606    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n" event={"ID":"f6a1c4c9-a5ca-4992-b21b-41c774df7df4","Type":"ContainerDied","Data":"f43c4242bd90edce5531262cd98538551a8d1808cdcc7bde1be17d3de11a5f00"}
Nov 02 21:24:55.481643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:55.481646    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"
Nov 02 21:24:55.482076 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:24:55.481664    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f43c4242bd90edce5531262cd98538551a8d1808cdcc7bde1be17d3de11a5f00"
Nov 02 21:25:07.892241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:07.892204    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:25:10.717884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.717850    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"]
Nov 02 21:25:10.718374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.718359    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f6a1c4c9-a5ca-4992-b21b-41c774df7df4" containerName="revoke-keys"
Nov 02 21:25:10.718418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.718375    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a1c4c9-a5ca-4992-b21b-41c774df7df4" containerName="revoke-keys"
Nov 02 21:25:10.718451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.718424    4528 cpu_manager.go:414] "containerMap: removing container" podUID="feacee19-1354-434e-a316-f4f1bd8e41f6" containerName="revoke-keys"
Nov 02 21:25:10.718451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.718430    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feacee19-1354-434e-a316-f4f1bd8e41f6" containerName="revoke-keys"
Nov 02 21:25:10.718510 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.718495    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a1c4c9-a5ca-4992-b21b-41c774df7df4" containerName="revoke-keys"
Nov 02 21:25:10.718510 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.718505    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="feacee19-1354-434e-a316-f4f1bd8e41f6" containerName="revoke-keys"
Nov 02 21:25:10.723080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.723066    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.775169 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.775141    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"]
Nov 02 21:25:10.821815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.821782    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-da55a9e52d6f\""
Nov 02 21:25:10.821815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.821809    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 21:25:10.821815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.821791    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 21:25:10.822663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.822642    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 21:25:10.825592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.825568    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 21:25:10.825592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.825576    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-941bbc45395a\""
Nov 02 21:25:10.825711 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.825580    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-884fm\""
Nov 02 21:25:10.875266 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.875238    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-plugins-config-volume\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.875393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.875356    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-routing-overlay-volume\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.875437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.875422    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.875471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.875460    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnb8k\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-kube-api-access-jnb8k\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.976204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.976105    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-routing-overlay-volume\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.976368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.976235    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.976530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.976493    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnb8k\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-kube-api-access-jnb8k\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.976611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.976590    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-plugins-config-volume\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.976934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.976899    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-routing-overlay-volume\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.977311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.977288    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-plugins-config-volume\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:10.978879 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:10.978863    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:11.057468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:11.053584    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnb8k\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-kube-api-access-jnb8k\") pod \"payload-processing-external-model-5c9f876fdf-ww9t2\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") " pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:11.335481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:11.335440    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:11.717799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:11.717765    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"]
Nov 02 21:25:11.717934 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:25:11.717834    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3365d6_8385_4b1c_a8b6_3703bc554ac0.slice/crio-e0ea5f941fd387a54d8daaf5294a19cab6d41734e2476d57d7800157b0a51c10 WatchSource:0}: Error finding container e0ea5f941fd387a54d8daaf5294a19cab6d41734e2476d57d7800157b0a51c10: Status 404 returned error can't find the container with id e0ea5f941fd387a54d8daaf5294a19cab6d41734e2476d57d7800157b0a51c10
Nov 02 21:25:12.573371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:12.573333    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2" event={"ID":"be3365d6-8385-4b1c-a8b6-3703bc554ac0","Type":"ContainerStarted","Data":"c5fd151ef0dc499ad935f9809bada36b618c3b7cdd1aeaa81046a5b471b388c8"}
Nov 02 21:25:12.573555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:12.573400    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:12.573555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:12.573423    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2" event={"ID":"be3365d6-8385-4b1c-a8b6-3703bc554ac0","Type":"ContainerStarted","Data":"e0ea5f941fd387a54d8daaf5294a19cab6d41734e2476d57d7800157b0a51c10"}
Nov 02 21:25:12.690042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:12.689978    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2" podStartSLOduration=2.6899648000000003 podStartE2EDuration="2.6899648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:12.687780383 +0000 UTC m=+999.951316703" watchObservedRunningTime="2026-11-02 21:25:12.6899648 +0000 UTC m=+999.953501133"
Nov 02 21:25:14.114305 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:25:14.114270    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:25:23.582546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:23.582516    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:25:29.401103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.401066    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"]
Nov 02 21:25:29.405093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.405074    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.458072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.458043    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"]
Nov 02 21:25:29.505314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.505280    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fksj\" (UniqueName: \"kubernetes.io/projected/9309246b-b945-4d1a-825d-c206032bca75-kube-api-access-4fksj\") pod \"maas-api-revoke-sub-dc4a4142-plrl4\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.505482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.505324    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9309246b-b945-4d1a-825d-c206032bca75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc4a4142-plrl4\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.505482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.505440    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9309246b-b945-4d1a-825d-c206032bca75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc4a4142-plrl4\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.505836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.505810    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:25:29.505836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.505834    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:25:29.505997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.505839    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:25:29.507540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.507523    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:25:29.606812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.606778    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fksj\" (UniqueName: \"kubernetes.io/projected/9309246b-b945-4d1a-825d-c206032bca75-kube-api-access-4fksj\") pod \"maas-api-revoke-sub-dc4a4142-plrl4\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.607017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.606839    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9309246b-b945-4d1a-825d-c206032bca75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc4a4142-plrl4\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.607017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.606882    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9309246b-b945-4d1a-825d-c206032bca75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc4a4142-plrl4\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.607696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.607666    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9309246b-b945-4d1a-825d-c206032bca75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc4a4142-plrl4\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.610014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.609983    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9309246b-b945-4d1a-825d-c206032bca75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc4a4142-plrl4\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.664083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.664008    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fksj\" (UniqueName: \"kubernetes.io/projected/9309246b-b945-4d1a-825d-c206032bca75-kube-api-access-4fksj\") pod \"maas-api-revoke-sub-dc4a4142-plrl4\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:29.717863 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:29.717833    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:30.087245 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:25:30.087216    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9309246b_b945_4d1a_825d_c206032bca75.slice/crio-ab8f97e014645da1327c98e60e2dcce562a75744d5501e8589335c6d0eba8092 WatchSource:0}: Error finding container ab8f97e014645da1327c98e60e2dcce562a75744d5501e8589335c6d0eba8092: Status 404 returned error can't find the container with id ab8f97e014645da1327c98e60e2dcce562a75744d5501e8589335c6d0eba8092
Nov 02 21:25:30.088970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:30.088929    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"]
Nov 02 21:25:30.685557 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:30.685460    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4" event={"ID":"9309246b-b945-4d1a-825d-c206032bca75","Type":"ContainerStarted","Data":"e9d3cbe856cb98feeb9749e0af1fd6586e6e409ac8c276bca107ac2685c57656"}
Nov 02 21:25:30.685557 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:30.685551    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4" event={"ID":"9309246b-b945-4d1a-825d-c206032bca75","Type":"ContainerStarted","Data":"ab8f97e014645da1327c98e60e2dcce562a75744d5501e8589335c6d0eba8092"}
Nov 02 21:25:30.798378 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:30.798326    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4" podStartSLOduration=1.7983138269999999 podStartE2EDuration="1.798313827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:30.798156046 +0000 UTC m=+1018.061692361" watchObservedRunningTime="2026-11-02 21:25:30.798313827 +0000 UTC m=+1018.061850143"
Nov 02 21:25:31.067668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.067638    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"]
Nov 02 21:25:31.071426 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.071410    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.121322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.121285    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.121490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.121337    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.121490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.121402    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.121490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.121433    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.121490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.121481    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.121736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.121560    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.121736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.121585    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.121736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.121633    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vx2z\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-kube-api-access-4vx2z\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.121736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.121672    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.126411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.126382    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"]
Nov 02 21:25:31.223213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223129    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223173    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223215    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vx2z\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-kube-api-access-4vx2z\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223251    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223302    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223329    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223389    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223421    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223471    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223762    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223785    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.223910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.223852    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.224024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.224003    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.224100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.224051    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.225429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.225408    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.225649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.225630    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.280251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.280226    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.280392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.280373    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vx2z\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-kube-api-access-4vx2z\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pd7dk\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.384303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.384269    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:31.691635 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.691600    4528 generic.go:356] "Generic (PLEG): container finished" podID="9309246b-b945-4d1a-825d-c206032bca75" containerID="e9d3cbe856cb98feeb9749e0af1fd6586e6e409ac8c276bca107ac2685c57656" exitCode=0
Nov 02 21:25:31.692016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.691687    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4" event={"ID":"9309246b-b945-4d1a-825d-c206032bca75","Type":"ContainerDied","Data":"e9d3cbe856cb98feeb9749e0af1fd6586e6e409ac8c276bca107ac2685c57656"}
Nov 02 21:25:31.817382 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:25:31.817352    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32c3f43_49f5_412b_b226_f1d0ce064ac7.slice/crio-99301da0e2eca4afa5bd3074c405005efa6a9574016086957f5844e5eb21889c WatchSource:0}: Error finding container 99301da0e2eca4afa5bd3074c405005efa6a9574016086957f5844e5eb21889c: Status 404 returned error can't find the container with id 99301da0e2eca4afa5bd3074c405005efa6a9574016086957f5844e5eb21889c
Nov 02 21:25:31.824695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.824663    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:25:31.824787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.824735    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:25:31.824787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.824760    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:25:31.858476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:31.858444    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"]
Nov 02 21:25:32.698463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.698425    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" event={"ID":"b32c3f43-49f5-412b-b226-f1d0ce064ac7","Type":"ContainerStarted","Data":"21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b"}
Nov 02 21:25:32.698965 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.698480    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" event={"ID":"b32c3f43-49f5-412b-b226-f1d0ce064ac7","Type":"ContainerStarted","Data":"99301da0e2eca4afa5bd3074c405005efa6a9574016086957f5844e5eb21889c"}
Nov 02 21:25:32.889329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.889304    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:32.927131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.926055    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podStartSLOduration=1.926033214 podStartE2EDuration="1.926033214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:32.86456161 +0000 UTC m=+1020.128097940" watchObservedRunningTime="2026-11-02 21:25:32.926033214 +0000 UTC m=+1020.189569533"
Nov 02 21:25:32.941598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.941568    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9309246b-b945-4d1a-825d-c206032bca75-kube-api-access-4fksj\" (UniqueName: \"kubernetes.io/projected/9309246b-b945-4d1a-825d-c206032bca75-kube-api-access-4fksj\") pod \"9309246b-b945-4d1a-825d-c206032bca75\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") "
Nov 02 21:25:32.941774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.941607    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9309246b-b945-4d1a-825d-c206032bca75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9309246b-b945-4d1a-825d-c206032bca75-maas-api-xks-ca\") pod \"9309246b-b945-4d1a-825d-c206032bca75\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") "
Nov 02 21:25:32.941774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.941684    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9309246b-b945-4d1a-825d-c206032bca75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9309246b-b945-4d1a-825d-c206032bca75-maas-api-openshift-ca\") pod \"9309246b-b945-4d1a-825d-c206032bca75\" (UID: \"9309246b-b945-4d1a-825d-c206032bca75\") "
Nov 02 21:25:32.942080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.942054    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9309246b-b945-4d1a-825d-c206032bca75-maas-api-openshift-ca" pod "9309246b-b945-4d1a-825d-c206032bca75" (UID: "9309246b-b945-4d1a-825d-c206032bca75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:25:32.943576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.943554    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9309246b-b945-4d1a-825d-c206032bca75-maas-api-xks-ca" pod "9309246b-b945-4d1a-825d-c206032bca75" (UID: "9309246b-b945-4d1a-825d-c206032bca75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:25:32.943651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:32.943601    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9309246b-b945-4d1a-825d-c206032bca75-kube-api-access-4fksj" pod "9309246b-b945-4d1a-825d-c206032bca75" (UID: "9309246b-b945-4d1a-825d-c206032bca75"). InnerVolumeSpecName "kube-api-access-4fksj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:25:33.043374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:33.043340    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fksj\" (UniqueName: \"kubernetes.io/projected/9309246b-b945-4d1a-825d-c206032bca75-kube-api-access-4fksj\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:25:33.043374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:33.043375    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9309246b-b945-4d1a-825d-c206032bca75-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:25:33.043579 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:33.043391    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9309246b-b945-4d1a-825d-c206032bca75-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:25:33.384792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:33.384762    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:33.386646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:33.386617    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 21:25:33.386802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:33.386665    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 21:25:33.722085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:33.721996    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"
Nov 02 21:25:33.726706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:33.722259    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4" event={"ID":"9309246b-b945-4d1a-825d-c206032bca75","Type":"ContainerDied","Data":"ab8f97e014645da1327c98e60e2dcce562a75744d5501e8589335c6d0eba8092"}
Nov 02 21:25:33.726782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:33.726715    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8f97e014645da1327c98e60e2dcce562a75744d5501e8589335c6d0eba8092"
Nov 02 21:25:34.385646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:34.385610    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 21:25:34.385821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:34.385671    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 21:25:35.385359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:35.385317    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 21:25:35.385730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:35.385377    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 21:25:36.385688 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:36.385652    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 21:25:36.386086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:36.385708    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 21:25:37.384684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.384647    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 21:25:37.384843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.384711    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 21:25:37.896010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.895972    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-t9vq9-s6hbq"]
Nov 02 21:25:37.896457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.896444    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9309246b-b945-4d1a-825d-c206032bca75" containerName="revoke-keys"
Nov 02 21:25:37.896514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.896458    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9309246b-b945-4d1a-825d-c206032bca75" containerName="revoke-keys"
Nov 02 21:25:37.896547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.896527    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9309246b-b945-4d1a-825d-c206032bca75" containerName="revoke-keys"
Nov 02 21:25:37.899881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.899867    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:37.955181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.955144    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t9vq9-s6hbq"]
Nov 02 21:25:37.994096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.994068    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/855c4070-9daa-49ec-9a92-317cd6b3acb1-archives-path\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:37.994282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.994260    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-trusted-ca-bundle\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:37.994331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.994303    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-service-ca-bundle\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:37.994376 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:37.994334    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbpwm\" (UniqueName: \"kubernetes.io/projected/855c4070-9daa-49ec-9a92-317cd6b3acb1-kube-api-access-wbpwm\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.001492 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.001468    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5jdjp\""
Nov 02 21:25:38.002599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.002583    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 21:25:38.095340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.095301    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/855c4070-9daa-49ec-9a92-317cd6b3acb1-archives-path\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.095534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.095443    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-trusted-ca-bundle\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.095534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.095476    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-service-ca-bundle\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.095534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.095501    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbpwm\" (UniqueName: \"kubernetes.io/projected/855c4070-9daa-49ec-9a92-317cd6b3acb1-kube-api-access-wbpwm\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.095704 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.095683    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/855c4070-9daa-49ec-9a92-317cd6b3acb1-archives-path\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.096027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.096010    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-service-ca-bundle\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.104079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.104057    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 21:25:38.106959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.106933    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-trusted-ca-bundle\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.155095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.155018    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbpwm\" (UniqueName: \"kubernetes.io/projected/855c4070-9daa-49ec-9a92-317cd6b3acb1-kube-api-access-wbpwm\") pod \"periodic-gathering-t9vq9-s6hbq\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") " pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.212160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.212132    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:25:38.385456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.385413    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 21:25:38.385637 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.385483    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 21:25:38.584914 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:25:38.584858    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855c4070_9daa_49ec_9a92_317cd6b3acb1.slice/crio-aad8c1091f989f9ec30832b26a17cad47880449f109dab37140e66477aa7c441 WatchSource:0}: Error finding container aad8c1091f989f9ec30832b26a17cad47880449f109dab37140e66477aa7c441: Status 404 returned error can't find the container with id aad8c1091f989f9ec30832b26a17cad47880449f109dab37140e66477aa7c441
Nov 02 21:25:38.585658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.585570    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t9vq9-s6hbq"]
Nov 02 21:25:38.667342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.667307    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:25:38.754671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.754633    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t9vq9-s6hbq" event={"ID":"855c4070-9daa-49ec-9a92-317cd6b3acb1","Type":"ContainerStarted","Data":"c0a2633141d094c20ce432dbb69cc332e35c3972f8da119ff67dd1b1158cdaae"}
Nov 02 21:25:38.754819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.754698    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t9vq9-s6hbq" event={"ID":"855c4070-9daa-49ec-9a92-317cd6b3acb1","Type":"ContainerStarted","Data":"aad8c1091f989f9ec30832b26a17cad47880449f109dab37140e66477aa7c441"}
Nov 02 21:25:38.873307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:38.873195    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-t9vq9-s6hbq" podStartSLOduration=1.873179605 podStartE2EDuration="1.873179605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:38.870504921 +0000 UTC m=+1026.134041265" watchObservedRunningTime="2026-11-02 21:25:38.873179605 +0000 UTC m=+1026.136715922"
Nov 02 21:25:39.385037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:39.384992    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 21:25:39.385656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:39.385068    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 21:25:41.385595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:41.385559    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:25:41.386089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:41.385636    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:25:42.386300 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:42.386261    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:25:42.386719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:42.386316    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:25:43.377603 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:43.377553    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:43.378340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:43.378317    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:43.379476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:43.379458    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:25:44.113502 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:25:44.113468    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:25:46.075640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.075606    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"]
Nov 02 21:25:46.080424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.080407    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.132852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.132820    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.133058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.132862    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.133058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.132947    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.133058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.133037    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.133234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.133135    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74ae5c-80fe-434b-9313-e42f60ae0518-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.133234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.133176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.133234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.133196    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.133338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.133309    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.133338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.133325    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx9pb\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-kube-api-access-rx9pb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.198079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.198040    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"]
Nov 02 21:25:46.234658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.234628    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.234658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.234662    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.234887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.234686    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.234887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.234712    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.234887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.234778    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74ae5c-80fe-434b-9313-e42f60ae0518-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.234887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.234830    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.234887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.234858    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.235162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.234906    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.235162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.234929    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx9pb\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-kube-api-access-rx9pb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.235298 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.235276    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.235371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.235323    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.235425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.235364    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.235563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.235544    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74ae5c-80fe-434b-9313-e42f60ae0518-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.235664 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.235649    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.237976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.237949    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.238085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.238007    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.295196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.295160    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.295392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.295372    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx9pb\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-kube-api-access-rx9pb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6js7r\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.394137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.394030    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:46.812958 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:25:46.812913    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d74ae5c_80fe_434b_9313_e42f60ae0518.slice/crio-766173c76bd8f1d802f9b13fed96e37d8fc7b2b5b1f45ecdc104572e2a595ff1 WatchSource:0}: Error finding container 766173c76bd8f1d802f9b13fed96e37d8fc7b2b5b1f45ecdc104572e2a595ff1: Status 404 returned error can't find the container with id 766173c76bd8f1d802f9b13fed96e37d8fc7b2b5b1f45ecdc104572e2a595ff1
Nov 02 21:25:46.815528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.815494    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:25:46.815660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.815560    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:25:46.815660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.815589    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:25:46.815761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.815741    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"]
Nov 02 21:25:46.854350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:46.854320    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" event={"ID":"0d74ae5c-80fe-434b-9313-e42f60ae0518","Type":"ContainerStarted","Data":"766173c76bd8f1d802f9b13fed96e37d8fc7b2b5b1f45ecdc104572e2a595ff1"}
Nov 02 21:25:47.860691 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:47.860657    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" event={"ID":"0d74ae5c-80fe-434b-9313-e42f60ae0518","Type":"ContainerStarted","Data":"ab0b54b75601731a7378fecdd44b2d116881189288c8623f67428a760de76e98"}
Nov 02 21:25:48.093188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:48.093129    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" podStartSLOduration=2.0931044659999998 podStartE2EDuration="2.093104466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:48.041059634 +0000 UTC m=+1035.304595952" watchObservedRunningTime="2026-11-02 21:25:48.093104466 +0000 UTC m=+1035.356640784"
Nov 02 21:25:48.395033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:48.394998    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:48.396585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:48.396562    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6js7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 21:25:48.396709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:48.396604    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 21:25:49.397559 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:49.397518    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6js7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 21:25:49.398058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:49.397582    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 21:25:50.395287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:50.395256    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6js7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 21:25:50.395446 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:50.395306    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 21:25:51.394766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:51.394726    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6js7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 21:25:51.395268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:51.394796    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 21:25:52.395221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:52.395185    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6js7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 21:25:52.395587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:52.395240    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 21:25:53.394577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:53.394537    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6js7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 21:25:53.394781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:53.394590    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 21:25:54.394525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:54.394490    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6js7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 21:25:54.395045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:54.394542    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 21:25:55.421304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:55.421268    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:55.422102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:55.422078    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:25:55.422881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:25:55.422861    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:26:01.216647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.216615    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-79js9"]
Nov 02 21:26:01.220565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.220549    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.246506 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.246478    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-routing-overlay-volume\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.246658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.246516    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8b6g\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-kube-api-access-z8b6g\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.246658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.246537    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-plugins-config-volume\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.246780 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.246724    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-provider-credentials\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.274181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.274140    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-79js9"]
Nov 02 21:26:01.348353 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.348318    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-provider-credentials\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.348548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.348419    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-routing-overlay-volume\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.348548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.348447    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8b6g\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-kube-api-access-z8b6g\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.348548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.348471    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-plugins-config-volume\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.350233 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.349551    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-plugins-config-volume\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.350557 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.350507    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-routing-overlay-volume\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.359781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.359761    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-provider-credentials\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.407839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.407812    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8b6g\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-kube-api-access-z8b6g\") pod \"payload-processing-external-model-84f575b468-79js9\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.533774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.533748    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:01.901864 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:26:01.901806    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e30810_dafc_45d3_9ce4_7f561964fe31.slice/crio-cf5aaca91b53595b62480791853343d7626cfc905cc762cb8d86ae1b71bf383f WatchSource:0}: Error finding container cf5aaca91b53595b62480791853343d7626cfc905cc762cb8d86ae1b71bf383f: Status 404 returned error can't find the container with id cf5aaca91b53595b62480791853343d7626cfc905cc762cb8d86ae1b71bf383f
Nov 02 21:26:01.906533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.906494    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-79js9"]
Nov 02 21:26:01.943732 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:01.943706    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9" event={"ID":"f0e30810-dafc-45d3-9ce4-7f561964fe31","Type":"ContainerStarted","Data":"cf5aaca91b53595b62480791853343d7626cfc905cc762cb8d86ae1b71bf383f"}
Nov 02 21:26:02.951977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:02.951942    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9" event={"ID":"f0e30810-dafc-45d3-9ce4-7f561964fe31","Type":"ContainerStarted","Data":"af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15"}
Nov 02 21:26:02.952385 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:02.952024    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:03.069351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:03.069297    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9" podStartSLOduration=2.069281798 podStartE2EDuration="2.069281798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:03.066521901 +0000 UTC m=+1050.330058221" watchObservedRunningTime="2026-11-02 21:26:03.069281798 +0000 UTC m=+1050.332818115"
Nov 02 21:26:05.948602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.948571    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"]
Nov 02 21:26:05.954441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.954417    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:05.995367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.995334    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr9n6\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-kube-api-access-tr9n6\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:05.995535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.995407    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409ba5bd-00c3-401d-bbda-e6191877887f-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:05.995535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.995451    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:05.995535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.995471    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:05.995634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.995610    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409ba5bd-00c3-401d-bbda-e6191877887f-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:05.995678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.995662    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:05.995770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.995750    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:05.995808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.995783    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:05.995852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:05.995809    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.060119 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.060074    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"]
Nov 02 21:26:06.096578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.096545    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9n6\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-kube-api-access-tr9n6\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.096764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.096629    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409ba5bd-00c3-401d-bbda-e6191877887f-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.096764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.096677    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.096764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.096704    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.096764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.096727    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409ba5bd-00c3-401d-bbda-e6191877887f-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.096764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.096756    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.097026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.096822    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.097026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.096845    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.097026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.096881    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.097403 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.097376    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.097510 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.097422    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.097663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.097641    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409ba5bd-00c3-401d-bbda-e6191877887f-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.097821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.097754    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.097937 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.097813    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.099713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.099690    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409ba5bd-00c3-401d-bbda-e6191877887f-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.101319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.101299    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.153943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.153911    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.154102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.153911    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9n6\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-kube-api-access-tr9n6\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.271036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.271011    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:06.706922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.706891    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"]
Nov 02 21:26:06.708622 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:26:06.708597    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409ba5bd_00c3_401d_bbda_e6191877887f.slice/crio-6eff04b7122963046fd0c7b9773082dcd7773b06322578297797c0037a83f83f WatchSource:0}: Error finding container 6eff04b7122963046fd0c7b9773082dcd7773b06322578297797c0037a83f83f: Status 404 returned error can't find the container with id 6eff04b7122963046fd0c7b9773082dcd7773b06322578297797c0037a83f83f
Nov 02 21:26:06.711062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.711031    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:26:06.711168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.711095    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:26:06.711168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.711153    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:26:06.995642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.995086    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" event={"ID":"409ba5bd-00c3-401d-bbda-e6191877887f","Type":"ContainerStarted","Data":"942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224"}
Nov 02 21:26:06.995642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:06.995343    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" event={"ID":"409ba5bd-00c3-401d-bbda-e6191877887f","Type":"ContainerStarted","Data":"6eff04b7122963046fd0c7b9773082dcd7773b06322578297797c0037a83f83f"}
Nov 02 21:26:07.121085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:07.121027    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podStartSLOduration=2.121008159 podStartE2EDuration="2.121008159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:07.120447222 +0000 UTC m=+1054.383983573" watchObservedRunningTime="2026-11-02 21:26:07.121008159 +0000 UTC m=+1054.384544483"
Nov 02 21:26:07.271624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:07.271529    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:07.274852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:07.274524    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 21:26:07.274852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:07.274581    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 21:26:08.271518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:08.271473    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 21:26:08.271991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:08.271531    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 21:26:09.271836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:09.271761    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 21:26:09.272314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:09.271836    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 21:26:10.232132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:10.229282    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:26:10.272310 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:10.272274    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 21:26:10.272713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:10.272337    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 21:26:11.274930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:11.274893    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 21:26:11.275466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:11.274953    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 21:26:12.271987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:12.271942    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 21:26:12.272219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:12.272018    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 21:26:13.287134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:13.271391    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 21:26:13.287742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:13.287703    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 21:26:13.631363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:13.631238    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6qkd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:26:13.631660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:13.631627    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:26:14.282135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:14.281497    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9" podUID="f0e30810-dafc-45d3-9ce4-7f561964fe31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:26:14.441254 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:14.441209    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pd7dk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:26:14.441845 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:14.441812    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:26:14.661198 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:14.660680    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:26:15.074422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:15.074378    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:15.075284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:15.075262    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:15.075855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:15.075836    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:26:20.834186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:20.834139    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-79js9"]
Nov 02 21:26:20.835525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:20.835492    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9" podUID="f0e30810-dafc-45d3-9ce4-7f561964fe31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:26:25.835410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:25.835339    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9" podUID="f0e30810-dafc-45d3-9ce4-7f561964fe31" containerName="payload-processing" containerID="cri-o://af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15" gracePeriod=25
Nov 02 21:26:26.238211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.238160    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:26.350930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.350323    4528 generic.go:356] "Generic (PLEG): container finished" podID="f0e30810-dafc-45d3-9ce4-7f561964fe31" containerID="af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15" exitCode=0
Nov 02 21:26:26.350930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.350539    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9" event={"ID":"f0e30810-dafc-45d3-9ce4-7f561964fe31","Type":"ContainerDied","Data":"af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15"}
Nov 02 21:26:26.350930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.350593    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9" event={"ID":"f0e30810-dafc-45d3-9ce4-7f561964fe31","Type":"ContainerDied","Data":"cf5aaca91b53595b62480791853343d7626cfc905cc762cb8d86ae1b71bf383f"}
Nov 02 21:26:26.350930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.350623    4528 scope.go:122] "RemoveContainer" containerID="af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15"
Nov 02 21:26:26.350930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.350792    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84f575b468-79js9"
Nov 02 21:26:26.359727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.359701    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-provider-credentials\") pod \"f0e30810-dafc-45d3-9ce4-7f561964fe31\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") "
Nov 02 21:26:26.359981 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.359947    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-routing-overlay-volume\") pod \"f0e30810-dafc-45d3-9ce4-7f561964fe31\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") "
Nov 02 21:26:26.360248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.360222    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-plugins-config-volume\") pod \"f0e30810-dafc-45d3-9ce4-7f561964fe31\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") "
Nov 02 21:26:26.360501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.360478    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-kube-api-access-z8b6g\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-kube-api-access-z8b6g\") pod \"f0e30810-dafc-45d3-9ce4-7f561964fe31\" (UID: \"f0e30810-dafc-45d3-9ce4-7f561964fe31\") "
Nov 02 21:26:26.375180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.372517    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-plugins-config-volume" pod "f0e30810-dafc-45d3-9ce4-7f561964fe31" (UID: "f0e30810-dafc-45d3-9ce4-7f561964fe31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:26:26.375581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.375524    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-routing-overlay-volume" pod "f0e30810-dafc-45d3-9ce4-7f561964fe31" (UID: "f0e30810-dafc-45d3-9ce4-7f561964fe31"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:26:26.402222 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.400717    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-kube-api-access-z8b6g" pod "f0e30810-dafc-45d3-9ce4-7f561964fe31" (UID: "f0e30810-dafc-45d3-9ce4-7f561964fe31"). InnerVolumeSpecName "kube-api-access-z8b6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:26:26.402222 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.400809    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-provider-credentials" pod "f0e30810-dafc-45d3-9ce4-7f561964fe31" (UID: "f0e30810-dafc-45d3-9ce4-7f561964fe31"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:26:26.488011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.487868    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8b6g\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-kube-api-access-z8b6g\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:26.488011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.487900    4528 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0e30810-dafc-45d3-9ce4-7f561964fe31-provider-credentials\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:26.488011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.487916    4528 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-routing-overlay-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:26.488011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.487930    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e30810-dafc-45d3-9ce4-7f561964fe31-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:26.488852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.488500    4528 scope.go:122] "RemoveContainer" containerID="af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15"
Nov 02 21:26:26.489199 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:26.489077    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15\": container with ID starting with af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15 not found: ID does not exist" containerID="af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15"
Nov 02 21:26:26.489199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.489135    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15"} err="failed to get container status \"af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15\": rpc error: code = NotFound desc = could not find container \"af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15\": container with ID starting with af20642a2040dcf11eb090bd81fed2f676270550d3935fea4706a8737f609f15 not found: ID does not exist"
Nov 02 21:26:26.801265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.801230    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-79js9"]
Nov 02 21:26:26.805686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.805655    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-79js9"]
Nov 02 21:26:26.918067 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:26.918031    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e30810-dafc-45d3-9ce4-7f561964fe31" path="/var/lib/kubelet/pods/f0e30810-dafc-45d3-9ce4-7f561964fe31/volumes"
Nov 02 21:26:28.626313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:28.626275    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6qkd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:26:28.626975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:28.626939    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:26:28.940813 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:28.940721    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-da55a9e52d6f: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:28.944187 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:28.944165    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:28.944391 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:28.944377    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials podName:be3365d6-8385-4b1c-a8b6-3703bc554ac0 nodeName:}" failed. No retries permitted until 2026-11-02 21:26:29.444356108 +0000 UTC m=+1076.707892424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials") pod "payload-processing-external-model-5c9f876fdf-ww9t2" (UID: "be3365d6-8385-4b1c-a8b6-3703bc554ac0") : secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:29.446882 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:29.446848    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-da55a9e52d6f: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:29.446882 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:29.446877    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:29.447085 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:29.446941    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials podName:be3365d6-8385-4b1c-a8b6-3703bc554ac0 nodeName:}" failed. No retries permitted until 2026-11-02 21:26:30.446926367 +0000 UTC m=+1077.710462693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials") pod "payload-processing-external-model-5c9f876fdf-ww9t2" (UID: "be3365d6-8385-4b1c-a8b6-3703bc554ac0") : secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:30.455084 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:30.455056    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-da55a9e52d6f: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:30.455084 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:30.455080    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:30.455608 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:30.455156    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials podName:be3365d6-8385-4b1c-a8b6-3703bc554ac0 nodeName:}" failed. No retries permitted until 2026-11-02 21:26:32.455137415 +0000 UTC m=+1079.718673711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials") pod "payload-processing-external-model-5c9f876fdf-ww9t2" (UID: "be3365d6-8385-4b1c-a8b6-3703bc554ac0") : secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:32.477945 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:32.477905    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-da55a9e52d6f: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:32.477945 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:32.477935    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:32.478511 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:32.477999    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials podName:be3365d6-8385-4b1c-a8b6-3703bc554ac0 nodeName:}" failed. No retries permitted until 2026-11-02 21:26:36.477979865 +0000 UTC m=+1083.741516164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials") pod "payload-processing-external-model-5c9f876fdf-ww9t2" (UID: "be3365d6-8385-4b1c-a8b6-3703bc554ac0") : secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:33.312436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:33.312361    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"]
Nov 02 21:26:36.522480 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:36.522450    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-da55a9e52d6f: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:36.522480 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:36.522475    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2: secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:36.522910 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:36.522526    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials podName:be3365d6-8385-4b1c-a8b6-3703bc554ac0 nodeName:}" failed. No retries permitted until 2026-11-02 21:26:44.522511623 +0000 UTC m=+1091.786047918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials") pod "payload-processing-external-model-5c9f876fdf-ww9t2" (UID: "be3365d6-8385-4b1c-a8b6-3703bc554ac0") : secret "agc-cred-da55a9e52d6f" not found
Nov 02 21:26:36.749307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.749274    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"]
Nov 02 21:26:36.749754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.749739    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f0e30810-dafc-45d3-9ce4-7f561964fe31" containerName="payload-processing"
Nov 02 21:26:36.749796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.749756    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e30810-dafc-45d3-9ce4-7f561964fe31" containerName="payload-processing"
Nov 02 21:26:36.749847 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.749838    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e30810-dafc-45d3-9ce4-7f561964fe31" containerName="payload-processing"
Nov 02 21:26:36.754522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.754506    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.806312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.806237    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"]
Nov 02 21:26:36.825493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.825467    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tls-certs\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.825629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.825522    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hm9\" (UniqueName: \"kubernetes.io/projected/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-kube-api-access-55hm9\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.825721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.825695    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-dshm\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.825765 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.825749    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-model-cache\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.825800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.825787    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tmp-dir\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.825845 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.825822    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-home\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.853664 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.853635    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"e2e-conflict-b-00e30839-kserve-self-signed-certs\""
Nov 02 21:26:36.939136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.931715    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hm9\" (UniqueName: \"kubernetes.io/projected/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-kube-api-access-55hm9\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.939595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.939551    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-dshm\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.939779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.939763    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-model-cache\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.939951 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.939934    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tmp-dir\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.942851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.941767    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-model-cache\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.942851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.942806    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tmp-dir\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.946429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.946401    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-home\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.947210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.947185    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tls-certs\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.947942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.946940    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-home\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.955778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.955407    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tls-certs\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:36.955778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:36.955585    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-dshm\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:37.006293 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:37.006263    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hm9\" (UniqueName: \"kubernetes.io/projected/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-kube-api-access-55hm9\") pod \"e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:37.066589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:37.066514    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:37.490312 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:26:37.490268    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecf6b457_e44c_4ea8_aa0e_e3afb024b1f7.slice/crio-7c9cd6edd1a64f690fcf7a4bdbf1b3273ccbcff4fe11d6f2ebbf21b5b9561b54 WatchSource:0}: Error finding container 7c9cd6edd1a64f690fcf7a4bdbf1b3273ccbcff4fe11d6f2ebbf21b5b9561b54: Status 404 returned error can't find the container with id 7c9cd6edd1a64f690fcf7a4bdbf1b3273ccbcff4fe11d6f2ebbf21b5b9561b54
Nov 02 21:26:37.492838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:37.492804    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"]
Nov 02 21:26:38.313073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.313022    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2" podUID="be3365d6-8385-4b1c-a8b6-3703bc554ac0" containerName="payload-processing" containerID="cri-o://c5fd151ef0dc499ad935f9809bada36b618c3b7cdd1aeaa81046a5b471b388c8" gracePeriod=25
Nov 02 21:26:38.474210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.474144    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7" event={"ID":"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7","Type":"ContainerStarted","Data":"0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006"}
Nov 02 21:26:38.474391 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.474218    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7" event={"ID":"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7","Type":"ContainerStarted","Data":"7c9cd6edd1a64f690fcf7a4bdbf1b3273ccbcff4fe11d6f2ebbf21b5b9561b54"}
Nov 02 21:26:38.478585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.478557    4528 generic.go:356] "Generic (PLEG): container finished" podID="be3365d6-8385-4b1c-a8b6-3703bc554ac0" containerID="c5fd151ef0dc499ad935f9809bada36b618c3b7cdd1aeaa81046a5b471b388c8" exitCode=0
Nov 02 21:26:38.478713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.478598    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2" event={"ID":"be3365d6-8385-4b1c-a8b6-3703bc554ac0","Type":"ContainerDied","Data":"c5fd151ef0dc499ad935f9809bada36b618c3b7cdd1aeaa81046a5b471b388c8"}
Nov 02 21:26:38.576329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.576268    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:26:38.592624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.592575    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7" podStartSLOduration=2.59255902 podStartE2EDuration="2.59255902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:38.590243173 +0000 UTC m=+1085.853779494" watchObservedRunningTime="2026-11-02 21:26:38.59255902 +0000 UTC m=+1085.856095337"
Nov 02 21:26:38.676651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.676619    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-routing-overlay-volume\") pod \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") "
Nov 02 21:26:38.676804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.676676    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials\") pod \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") "
Nov 02 21:26:38.676804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.676770    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-plugins-config-volume\") pod \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") "
Nov 02 21:26:38.676804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.676800    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-kube-api-access-jnb8k\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-kube-api-access-jnb8k\") pod \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\" (UID: \"be3365d6-8385-4b1c-a8b6-3703bc554ac0\") "
Nov 02 21:26:38.677083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.677001    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-routing-overlay-volume" pod "be3365d6-8385-4b1c-a8b6-3703bc554ac0" (UID: "be3365d6-8385-4b1c-a8b6-3703bc554ac0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:26:38.677243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.677222    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-plugins-config-volume" pod "be3365d6-8385-4b1c-a8b6-3703bc554ac0" (UID: "be3365d6-8385-4b1c-a8b6-3703bc554ac0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:26:38.678904 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.678884    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials" pod "be3365d6-8385-4b1c-a8b6-3703bc554ac0" (UID: "be3365d6-8385-4b1c-a8b6-3703bc554ac0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:26:38.679005 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.678946    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-kube-api-access-jnb8k" pod "be3365d6-8385-4b1c-a8b6-3703bc554ac0" (UID: "be3365d6-8385-4b1c-a8b6-3703bc554ac0"). InnerVolumeSpecName "kube-api-access-jnb8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:26:38.778541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.778508    4528 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-routing-overlay-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:38.778541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.778535    4528 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-provider-credentials\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:38.778541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.778545    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3365d6-8385-4b1c-a8b6-3703bc554ac0-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:38.778767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:38.778554    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnb8k\" (UniqueName: \"kubernetes.io/projected/be3365d6-8385-4b1c-a8b6-3703bc554ac0-kube-api-access-jnb8k\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:26:39.066798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:39.066761    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:39.485512 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:39.485479    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2" event={"ID":"be3365d6-8385-4b1c-a8b6-3703bc554ac0","Type":"ContainerDied","Data":"e0ea5f941fd387a54d8daaf5294a19cab6d41734e2476d57d7800157b0a51c10"}
Nov 02 21:26:39.485942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:39.485541    4528 scope.go:122] "RemoveContainer" containerID="c5fd151ef0dc499ad935f9809bada36b618c3b7cdd1aeaa81046a5b471b388c8"
Nov 02 21:26:39.485942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:39.485674    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"
Nov 02 21:26:39.655643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:39.655609    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"]
Nov 02 21:26:39.705818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:39.705786    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c9f876fdf-ww9t2"]
Nov 02 21:26:40.912285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:40.912255    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3365d6-8385-4b1c-a8b6-3703bc554ac0" path="/var/lib/kubelet/pods/be3365d6-8385-4b1c-a8b6-3703bc554ac0/volumes"
Nov 02 21:26:44.112983 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:26:44.112889    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:26:44.631993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:44.631961    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:26:47.081388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:47.080530    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:47.860130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:47.859492    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:48.338133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:48.337715    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:26:52.224578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.224533    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"]
Nov 02 21:26:52.225339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.225313    4528 cpu_manager.go:414] "containerMap: removing container" podUID="be3365d6-8385-4b1c-a8b6-3703bc554ac0" containerName="payload-processing"
Nov 02 21:26:52.225541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.225521    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3365d6-8385-4b1c-a8b6-3703bc554ac0" containerName="payload-processing"
Nov 02 21:26:52.225829 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.225814    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3365d6-8385-4b1c-a8b6-3703bc554ac0" containerName="payload-processing"
Nov 02 21:26:52.231340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.231313    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.338092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.338049    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"]
Nov 02 21:26:52.342301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.342267    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.342448 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.342410    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.342860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.342467    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.342860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.342498    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.342860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.342600    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.342860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.342646    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.342860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.342686    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.342860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.342727    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.342860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.342758    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-kube-api-access-cvtr8\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.443892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.443856    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.443912    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-kube-api-access-cvtr8\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444009    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444084    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444155    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444184    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444345 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444226    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444345 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444266    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444345 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444307    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444545    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444633    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.444773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444701    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.445027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.444894    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.445090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.445037    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.446500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.446472    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.446748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.446727    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.500987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.500891    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.501160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.501077    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-kube-api-access-cvtr8\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.547181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.547160    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:52.964311 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:26:52.964278    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e058f5_4c9d_41e7_a301_22e81d9270b2.slice/crio-7c4344ce5b953f537441849461b304b33196376550cbd8921112e8f1fbba2b8d WatchSource:0}: Error finding container 7c4344ce5b953f537441849461b304b33196376550cbd8921112e8f1fbba2b8d: Status 404 returned error can't find the container with id 7c4344ce5b953f537441849461b304b33196376550cbd8921112e8f1fbba2b8d
Nov 02 21:26:52.966913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.966872    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:26:52.966998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.966959    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:26:52.967036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.967019    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:26:52.967649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:52.967631    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"]
Nov 02 21:26:53.735968 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:53.735927    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" event={"ID":"a1e058f5-4c9d-41e7-a301-22e81d9270b2","Type":"ContainerStarted","Data":"a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529"}
Nov 02 21:26:53.736441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:53.735994    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" event={"ID":"a1e058f5-4c9d-41e7-a301-22e81d9270b2","Type":"ContainerStarted","Data":"7c4344ce5b953f537441849461b304b33196376550cbd8921112e8f1fbba2b8d"}
Nov 02 21:26:53.853405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:53.853355    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" podStartSLOduration=1.853339934 podStartE2EDuration="1.853339934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:53.851992848 +0000 UTC m=+1101.115529166" watchObservedRunningTime="2026-11-02 21:26:53.853339934 +0000 UTC m=+1101.116876252"
Nov 02 21:26:54.547576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:54.547538    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:54.549358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:54.549335    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 21:26:54.549482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:54.549381    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 21:26:55.548485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:55.548449    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 21:26:55.548895 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:55.548516    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 21:26:56.548330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:56.548296    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 21:26:56.548490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:56.548348    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 21:26:57.547502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:57.547469    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 21:26:57.548017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:57.547525    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 21:26:58.548155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:58.548092    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 21:26:58.548582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:58.548171    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 21:26:59.767309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:59.767268    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:59.768011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:59.767992    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:26:59.769310 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:26:59.769289    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:27:07.470082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.470047    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"]
Nov 02 21:27:07.473476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.473457    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.528640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.528540    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"]
Nov 02 21:27:07.574424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.574395    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"openshift-service-ca.crt\""
Nov 02 21:27:07.574587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.574416    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"kube-root-ca.crt\""
Nov 02 21:27:07.574587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.574483    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"default-dockercfg-tpxs7\""
Nov 02 21:27:07.576068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.576053    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"distinct-2-w1-5f765a-kserve-self-signed-certs\""
Nov 02 21:27:07.646496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.644758    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-home\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.646496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.644816    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-dshm\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.646496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.644910    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-model-cache\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.646496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.644989    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21efa86-2527-440b-acd0-657466f62f1e-tls-certs\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.646496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.645013    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gmk\" (UniqueName: \"kubernetes.io/projected/f21efa86-2527-440b-acd0-657466f62f1e-kube-api-access-m2gmk\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.646496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.645051    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-tmp-dir\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.746395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.746311    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21efa86-2527-440b-acd0-657466f62f1e-tls-certs\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.746395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.746345    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gmk\" (UniqueName: \"kubernetes.io/projected/f21efa86-2527-440b-acd0-657466f62f1e-kube-api-access-m2gmk\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.746395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.746369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-tmp-dir\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.746615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.746471    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-home\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.746615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.746511    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-dshm\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.746615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.746599    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-model-cache\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.746926 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.746900    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-tmp-dir\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.746926 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.746919    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-home\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.747056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.746962    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-model-cache\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.748582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.748565    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-dshm\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.748731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.748714    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21efa86-2527-440b-acd0-657466f62f1e-tls-certs\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:07.804988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:07.804941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gmk\" (UniqueName: \"kubernetes.io/projected/f21efa86-2527-440b-acd0-657466f62f1e-kube-api-access-m2gmk\") pod \"distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:08.086227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:08.086189    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:08.453714 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:27:08.453684    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf21efa86_2527_440b_acd0_657466f62f1e.slice/crio-73a2a5db99f5626b782a0e2ca1ced4a6dfe752dbfe135e479baa0802395899d2 WatchSource:0}: Error finding container 73a2a5db99f5626b782a0e2ca1ced4a6dfe752dbfe135e479baa0802395899d2: Status 404 returned error can't find the container with id 73a2a5db99f5626b782a0e2ca1ced4a6dfe752dbfe135e479baa0802395899d2
Nov 02 21:27:08.456659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:08.456632    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"]
Nov 02 21:27:08.822339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:08.822301    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" event={"ID":"f21efa86-2527-440b-acd0-657466f62f1e","Type":"ContainerStarted","Data":"1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e"}
Nov 02 21:27:08.822708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:08.822346    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" event={"ID":"f21efa86-2527-440b-acd0-657466f62f1e","Type":"ContainerStarted","Data":"73a2a5db99f5626b782a0e2ca1ced4a6dfe752dbfe135e479baa0802395899d2"}
Nov 02 21:27:08.937494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:08.937434    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podStartSLOduration=1.937416936 podStartE2EDuration="1.937416936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:27:08.935716115 +0000 UTC m=+1116.199252431" watchObservedRunningTime="2026-11-02 21:27:08.937416936 +0000 UTC m=+1116.200953255"
Nov 02 21:27:09.089726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:09.089621    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:14.112583 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:27:14.112544    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:27:17.874542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:17.874512    4528 generic.go:356] "Generic (PLEG): container finished" podID="855c4070-9daa-49ec-9a92-317cd6b3acb1" containerID="c0a2633141d094c20ce432dbb69cc332e35c3972f8da119ff67dd1b1158cdaae" exitCode=0
Nov 02 21:27:17.874985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:17.874579    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t9vq9-s6hbq" event={"ID":"855c4070-9daa-49ec-9a92-317cd6b3acb1","Type":"ContainerDied","Data":"c0a2633141d094c20ce432dbb69cc332e35c3972f8da119ff67dd1b1158cdaae"}
Nov 02 21:27:18.086365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:18.086338    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:18.104952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:18.104922    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:18.891082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:18.891055    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:27:19.026564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.026534    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:27:19.077335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077298    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/855c4070-9daa-49ec-9a92-317cd6b3acb1-kube-api-access-wbpwm\" (UniqueName: \"kubernetes.io/projected/855c4070-9daa-49ec-9a92-317cd6b3acb1-kube-api-access-wbpwm\") pod \"855c4070-9daa-49ec-9a92-317cd6b3acb1\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") "
Nov 02 21:27:19.077505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077354    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-service-ca-bundle\") pod \"855c4070-9daa-49ec-9a92-317cd6b3acb1\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") "
Nov 02 21:27:19.077505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077425    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-trusted-ca-bundle\") pod \"855c4070-9daa-49ec-9a92-317cd6b3acb1\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") "
Nov 02 21:27:19.077604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077544    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855c4070-9daa-49ec-9a92-317cd6b3acb1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/855c4070-9daa-49ec-9a92-317cd6b3acb1-archives-path\") pod \"855c4070-9daa-49ec-9a92-317cd6b3acb1\" (UID: \"855c4070-9daa-49ec-9a92-317cd6b3acb1\") "
Nov 02 21:27:19.077838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077724    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-service-ca-bundle" pod "855c4070-9daa-49ec-9a92-317cd6b3acb1" (UID: "855c4070-9daa-49ec-9a92-317cd6b3acb1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:27:19.077948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077811    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-trusted-ca-bundle" pod "855c4070-9daa-49ec-9a92-317cd6b3acb1" (UID: "855c4070-9daa-49ec-9a92-317cd6b3acb1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:27:19.077948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077852    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855c4070-9daa-49ec-9a92-317cd6b3acb1-archives-path" pod "855c4070-9daa-49ec-9a92-317cd6b3acb1" (UID: "855c4070-9daa-49ec-9a92-317cd6b3acb1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:27:19.078021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077960    4528 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/855c4070-9daa-49ec-9a92-317cd6b3acb1-archives-path\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:27:19.078021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077973    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-service-ca-bundle\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:27:19.078021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.077982    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/855c4070-9daa-49ec-9a92-317cd6b3acb1-trusted-ca-bundle\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:27:19.079414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.079392    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855c4070-9daa-49ec-9a92-317cd6b3acb1-kube-api-access-wbpwm" pod "855c4070-9daa-49ec-9a92-317cd6b3acb1" (UID: "855c4070-9daa-49ec-9a92-317cd6b3acb1"). InnerVolumeSpecName "kube-api-access-wbpwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:27:19.179006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.178925    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbpwm\" (UniqueName: \"kubernetes.io/projected/855c4070-9daa-49ec-9a92-317cd6b3acb1-kube-api-access-wbpwm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:27:19.886575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.886541    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t9vq9-s6hbq"
Nov 02 21:27:19.886782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.886576    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t9vq9-s6hbq" event={"ID":"855c4070-9daa-49ec-9a92-317cd6b3acb1","Type":"ContainerDied","Data":"aad8c1091f989f9ec30832b26a17cad47880449f109dab37140e66477aa7c441"}
Nov 02 21:27:19.886782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:19.886617    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad8c1091f989f9ec30832b26a17cad47880449f109dab37140e66477aa7c441"
Nov 02 21:27:21.621071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:21.621029    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:27:31.149201 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.149164    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"]
Nov 02 21:27:31.149824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.149809    4528 cpu_manager.go:414] "containerMap: removing container" podUID="855c4070-9daa-49ec-9a92-317cd6b3acb1" containerName="insights-gathering"
Nov 02 21:27:31.149867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.149826    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="855c4070-9daa-49ec-9a92-317cd6b3acb1" containerName="insights-gathering"
Nov 02 21:27:31.149990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.149977    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="855c4070-9daa-49ec-9a92-317cd6b3acb1" containerName="insights-gathering"
Nov 02 21:27:31.153844 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.153828    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.203655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.203616    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"]
Nov 02 21:27:31.226532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.226495    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-dshm\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.226532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.226532    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-home\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.226746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.226569    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-model-cache\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.226746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.226695    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tls-certs\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.226746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.226726    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tmp-dir\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.226849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.226828    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvnmd\" (UniqueName: \"kubernetes.io/projected/4dde1ed2-c163-4938-a9f2-46f39a4d257a-kube-api-access-vvnmd\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.250596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.250563    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"openshift-service-ca.crt\""
Nov 02 21:27:31.250596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.250564    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"kube-root-ca.crt\""
Nov 02 21:27:31.250801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.250625    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"distinct-w0-0c9014-kserve-self-signed-certs\""
Nov 02 21:27:31.256483 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.256461    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"default-dockercfg-c9nbm\""
Nov 02 21:27:31.338227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.336204    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-dshm\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.338227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.336250    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-home\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.338227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.336329    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-model-cache\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.338227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.336379    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tls-certs\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.338227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.336410    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tmp-dir\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.338227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.336520    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvnmd\" (UniqueName: \"kubernetes.io/projected/4dde1ed2-c163-4938-a9f2-46f39a4d257a-kube-api-access-vvnmd\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.338227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.337800    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-model-cache\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.338227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.338047    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-home\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.338894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.338776    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tmp-dir\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.355828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.355794    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tls-certs\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.357869 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.356231    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-dshm\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.394153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.394099    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvnmd\" (UniqueName: \"kubernetes.io/projected/4dde1ed2-c163-4938-a9f2-46f39a4d257a-kube-api-access-vvnmd\") pod \"distinct-w0-0c9014-kserve-55c58d586d-qt9lt\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.465737 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.465646    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:31.910904 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:27:31.910867    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dde1ed2_c163_4938_a9f2_46f39a4d257a.slice/crio-0e6ac341f6c13a756253f090c523fcab3cf48b92b51274d9ffd55d7c3f3778d9 WatchSource:0}: Error finding container 0e6ac341f6c13a756253f090c523fcab3cf48b92b51274d9ffd55d7c3f3778d9: Status 404 returned error can't find the container with id 0e6ac341f6c13a756253f090c523fcab3cf48b92b51274d9ffd55d7c3f3778d9
Nov 02 21:27:31.913585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.913561    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"]
Nov 02 21:27:31.958947 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:31.958914    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" event={"ID":"4dde1ed2-c163-4938-a9f2-46f39a4d257a","Type":"ContainerStarted","Data":"0e6ac341f6c13a756253f090c523fcab3cf48b92b51274d9ffd55d7c3f3778d9"}
Nov 02 21:27:32.966374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:32.966331    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" event={"ID":"4dde1ed2-c163-4938-a9f2-46f39a4d257a","Type":"ContainerStarted","Data":"8c6ae959acbb24a3a65d09dc1c070d707433117f6f6bf5bac3f17afaa453c25e"}
Nov 02 21:27:33.053312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:33.053279    4528 scope.go:122] "RemoveContainer" containerID="b520cb7b0c5e60635caf7c1c01f08a40d546abc368eb8799cb71579049afad2b"
Nov 02 21:27:33.062457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:33.062438    4528 scope.go:122] "RemoveContainer" containerID="7811e40bacac6f03b1bec63e1e4ea9dc99cee2bc92f8950aca367936ceb1bcf6"
Nov 02 21:27:33.070996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:33.070971    4528 scope.go:122] "RemoveContainer" containerID="7d5e377bdae0893233d71a3e090ea76cde7ab6d75a69f103693c564c8d88a3b3"
Nov 02 21:27:33.079408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:33.079387    4528 scope.go:122] "RemoveContainer" containerID="62c9a31c014dc0cb60a323cad64ce1f73f0f4dba7254713d60769f6fb573c6f2"
Nov 02 21:27:33.081396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:33.081345    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podStartSLOduration=2.081329519 podStartE2EDuration="2.081329519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:27:33.080028598 +0000 UTC m=+1140.343564913" watchObservedRunningTime="2026-11-02 21:27:33.081329519 +0000 UTC m=+1140.344865838"
Nov 02 21:27:33.087567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:33.087547    4528 scope.go:122] "RemoveContainer" containerID="21315ba463eab2e9c4475c8cba03435831531d6cf30439340d4d339746369e51"
Nov 02 21:27:33.466209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:33.466166    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:40.722962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.722929    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"]
Nov 02 21:27:40.728420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.728402    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.745239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.745212    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7pk4\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-kube-api-access-x7pk4\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.745382 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.745254    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.745451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.745378    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772fd287-d421-4a6e-86cb-c9ef846603df-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.745505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.745484    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.745552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.745514    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772fd287-d421-4a6e-86cb-c9ef846603df-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.745552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.745535    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.745649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.745637    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.745685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.745664    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.745730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.745684    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.781799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.781771    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f765a-openshift-default-dockercfg-6vjh4\""
Nov 02 21:27:40.834747 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.834716    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"]
Nov 02 21:27:40.846316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.846287    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.846316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.846318    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772fd287-d421-4a6e-86cb-c9ef846603df-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.846692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.846337    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.846692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.846377    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.846692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.846411    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.846692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.846442    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.846692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.846491    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7pk4\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-kube-api-access-x7pk4\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.846692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.846519    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.847020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.846784    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.847265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.847245    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772fd287-d421-4a6e-86cb-c9ef846603df-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.847487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.847468    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.847609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.847589    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.849643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.849602    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772fd287-d421-4a6e-86cb-c9ef846603df-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.849787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.849609    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.857408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.857380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.861024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.860998    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772fd287-d421-4a6e-86cb-c9ef846603df-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.903966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.903935    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:40.904133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:40.904024    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7pk4\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-kube-api-access-x7pk4\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:41.041440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:41.041401    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:41.462872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:41.462828    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"]
Nov 02 21:27:41.464446 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:41.464413    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:27:41.464535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:41.464476    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:27:41.464535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:41.464505    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:27:41.466086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:41.466060    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:41.478998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:41.478978    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:42.016559 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:42.016523    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" event={"ID":"772fd287-d421-4a6e-86cb-c9ef846603df","Type":"ContainerStarted","Data":"9756ca8039029b7db63320b9e768072739d204172e416a3aa0590b58accfa373"}
Nov 02 21:27:42.017035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:42.016580    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" event={"ID":"772fd287-d421-4a6e-86cb-c9ef846603df","Type":"ContainerStarted","Data":"8d297db23adb2e61eda32ad412fe71cda3ec0bfb98096d29df94970031261221"}
Nov 02 21:27:42.027749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:42.027726    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:27:42.041963 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:42.041942    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:42.043550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:42.043531    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 21:27:42.043631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:42.043569    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 21:27:42.135655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:42.135602    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podStartSLOduration=2.135588341 podStartE2EDuration="2.135588341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:27:42.13273915 +0000 UTC m=+1149.396275471" watchObservedRunningTime="2026-11-02 21:27:42.135588341 +0000 UTC m=+1149.399124657"
Nov 02 21:27:43.042142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:43.042094    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 21:27:43.042548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:43.042165    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 21:27:44.042258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:44.042226    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 21:27:44.042634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:44.042285    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 21:27:44.134926 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:27:44.134891    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:27:44.559774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:44.559734    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:27:45.049144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:45.046672    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 21:27:45.049144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:45.046732    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 21:27:46.042508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:46.042477    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 21:27:46.042717 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:46.042532    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 21:27:47.042486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:47.042451    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 21:27:47.042882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:47.042501    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 21:27:48.042173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:48.042139    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 21:27:48.042433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:48.042201    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 21:27:48.151097 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:48.151063    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"]
Nov 02 21:27:49.046749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:49.046716    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:49.047016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:49.046991    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:27:49.047721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:27:49.047704    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:28:03.171138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.171064    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7" podUID="ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" containerName="main" containerID="cri-o://0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006" gracePeriod=45
Nov 02 21:28:03.422940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.422887    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:28:03.503891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.503860    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-home\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-home\") pod \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") "
Nov 02 21:28:03.504052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.503907    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tmp-dir\") pod \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") "
Nov 02 21:28:03.504052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.503944    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-dshm\") pod \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") "
Nov 02 21:28:03.504052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.504014    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tls-certs\") pod \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") "
Nov 02 21:28:03.504052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.504050    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-model-cache\") pod \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") "
Nov 02 21:28:03.504285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.504141    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-kube-api-access-55hm9\" (UniqueName: \"kubernetes.io/projected/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-kube-api-access-55hm9\") pod \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\" (UID: \"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7\") "
Nov 02 21:28:03.504285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.504176    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-home" pod "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" (UID: "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:28:03.504285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.504212    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tmp-dir" pod "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" (UID: "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:28:03.504432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.504392    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-model-cache" pod "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" (UID: "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:28:03.504608 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.504591    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:03.504662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.504613    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:03.504662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.504627    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:03.506268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.506247    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-dshm" pod "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" (UID: "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:28:03.506367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.506348    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-kube-api-access-55hm9" pod "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" (UID: "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7"). InnerVolumeSpecName "kube-api-access-55hm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:28:03.506410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.506367    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tls-certs" pod "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" (UID: "ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:28:03.605746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.605708    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:03.605746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.605743    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55hm9\" (UniqueName: \"kubernetes.io/projected/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-kube-api-access-55hm9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:03.605746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:03.605755    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:04.196153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.196095    4528 generic.go:356] "Generic (PLEG): container finished" podID="ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" containerID="0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006" exitCode=0
Nov 02 21:28:04.196153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.196138    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7" event={"ID":"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7","Type":"ContainerDied","Data":"0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006"}
Nov 02 21:28:04.196153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.196150    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"
Nov 02 21:28:04.196650 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.196182    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7" event={"ID":"ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7","Type":"ContainerDied","Data":"7c9cd6edd1a64f690fcf7a4bdbf1b3273ccbcff4fe11d6f2ebbf21b5b9561b54"}
Nov 02 21:28:04.196650 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.196200    4528 scope.go:122] "RemoveContainer" containerID="0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006"
Nov 02 21:28:04.206237 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.206207    4528 scope.go:122] "RemoveContainer" containerID="0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006"
Nov 02 21:28:04.206499 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:28:04.206478    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006\": container with ID starting with 0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006 not found: ID does not exist" containerID="0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006"
Nov 02 21:28:04.206592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.206506    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006"} err="failed to get container status \"0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006\": rpc error: code = NotFound desc = could not find container \"0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006\": container with ID starting with 0820784b8890ebfcd0dcf7b8fefd3643ebc5c4fda45472dcc49a6235d5021006 not found: ID does not exist"
Nov 02 21:28:04.364094 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.364059    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"]
Nov 02 21:28:04.414318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.414287    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-b-00e30839-kserve-bc6fbfcf7-s2wt7"]
Nov 02 21:28:04.917645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:04.917603    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" path="/var/lib/kubelet/pods/ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7/volumes"
Nov 02 21:28:06.541436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.541395    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"]
Nov 02 21:28:06.542595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.542567    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" containerName="main"
Nov 02 21:28:06.542595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.542594    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" containerName="main"
Nov 02 21:28:06.542836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.542823    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf6b457-e44c-4ea8-aa0e-e3afb024b1f7" containerName="main"
Nov 02 21:28:06.548430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.548406    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.595844 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.595809    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"]
Nov 02 21:28:06.638804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.638772    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51483b5-2nq9b\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.639006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.638970    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkg4v\" (UniqueName: \"kubernetes.io/projected/efc23b07-2906-4a12-b45a-7064e832aedc-kube-api-access-tkg4v\") pod \"maas-api-revoke-sub-e51483b5-2nq9b\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.639156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.639136    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51483b5-2nq9b\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.642511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.642486    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:28:06.642631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.642513    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:28:06.642631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.642513    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:28:06.651301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.651283    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:28:06.740096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.740055    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkg4v\" (UniqueName: \"kubernetes.io/projected/efc23b07-2906-4a12-b45a-7064e832aedc-kube-api-access-tkg4v\") pod \"maas-api-revoke-sub-e51483b5-2nq9b\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.740296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.740184    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51483b5-2nq9b\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.740296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.740217    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51483b5-2nq9b\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.740807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.740784    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51483b5-2nq9b\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.742542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.742523    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51483b5-2nq9b\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.796792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.796735    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkg4v\" (UniqueName: \"kubernetes.io/projected/efc23b07-2906-4a12-b45a-7064e832aedc-kube-api-access-tkg4v\") pod \"maas-api-revoke-sub-e51483b5-2nq9b\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:06.861579 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:06.861551    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:07.230888 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:28:07.230850    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc23b07_2906_4a12_b45a_7064e832aedc.slice/crio-61e9a50b4d1faab389e779a6b62b1abda5fe29e7de529bb41e3f120f6f73049b WatchSource:0}: Error finding container 61e9a50b4d1faab389e779a6b62b1abda5fe29e7de529bb41e3f120f6f73049b: Status 404 returned error can't find the container with id 61e9a50b4d1faab389e779a6b62b1abda5fe29e7de529bb41e3f120f6f73049b
Nov 02 21:28:07.233416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:07.233393    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"]
Nov 02 21:28:08.221520 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:08.221488    4528 generic.go:356] "Generic (PLEG): container finished" podID="efc23b07-2906-4a12-b45a-7064e832aedc" containerID="c007c7a14e501ceee38b8051b479ef71760577bd4c7fb3948847f1ca9eb35c72" exitCode=0
Nov 02 21:28:08.221876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:08.221561    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b" event={"ID":"efc23b07-2906-4a12-b45a-7064e832aedc","Type":"ContainerDied","Data":"c007c7a14e501ceee38b8051b479ef71760577bd4c7fb3948847f1ca9eb35c72"}
Nov 02 21:28:08.221876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:08.221616    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b" event={"ID":"efc23b07-2906-4a12-b45a-7064e832aedc","Type":"ContainerStarted","Data":"61e9a50b4d1faab389e779a6b62b1abda5fe29e7de529bb41e3f120f6f73049b"}
Nov 02 21:28:09.372351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.372326    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:09.471408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.471374    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc23b07-2906-4a12-b45a-7064e832aedc-kube-api-access-tkg4v\" (UniqueName: \"kubernetes.io/projected/efc23b07-2906-4a12-b45a-7064e832aedc-kube-api-access-tkg4v\") pod \"efc23b07-2906-4a12-b45a-7064e832aedc\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") "
Nov 02 21:28:09.471578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.471481    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-openshift-ca\") pod \"efc23b07-2906-4a12-b45a-7064e832aedc\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") "
Nov 02 21:28:09.471578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.471530    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-xks-ca\") pod \"efc23b07-2906-4a12-b45a-7064e832aedc\" (UID: \"efc23b07-2906-4a12-b45a-7064e832aedc\") "
Nov 02 21:28:09.471816 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.471791    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-openshift-ca" pod "efc23b07-2906-4a12-b45a-7064e832aedc" (UID: "efc23b07-2906-4a12-b45a-7064e832aedc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:28:09.473535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.473514    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-xks-ca" pod "efc23b07-2906-4a12-b45a-7064e832aedc" (UID: "efc23b07-2906-4a12-b45a-7064e832aedc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:28:09.473619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.473512    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc23b07-2906-4a12-b45a-7064e832aedc-kube-api-access-tkg4v" pod "efc23b07-2906-4a12-b45a-7064e832aedc" (UID: "efc23b07-2906-4a12-b45a-7064e832aedc"). InnerVolumeSpecName "kube-api-access-tkg4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:28:09.572900 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.572862    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:09.572900 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.572897    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc23b07-2906-4a12-b45a-7064e832aedc-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:09.572900 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:09.572907    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkg4v\" (UniqueName: \"kubernetes.io/projected/efc23b07-2906-4a12-b45a-7064e832aedc-kube-api-access-tkg4v\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:10.235638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:10.235611    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"
Nov 02 21:28:10.235810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:10.235643    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b" event={"ID":"efc23b07-2906-4a12-b45a-7064e832aedc","Type":"ContainerDied","Data":"61e9a50b4d1faab389e779a6b62b1abda5fe29e7de529bb41e3f120f6f73049b"}
Nov 02 21:28:10.235810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:10.235692    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61e9a50b4d1faab389e779a6b62b1abda5fe29e7de529bb41e3f120f6f73049b"
Nov 02 21:28:14.112714 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:28:14.112678    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:28:19.203392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:19.203352    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:28:21.592537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.592504    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"]
Nov 02 21:28:21.593017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.593002    4528 cpu_manager.go:414] "containerMap: removing container" podUID="efc23b07-2906-4a12-b45a-7064e832aedc" containerName="revoke-keys"
Nov 02 21:28:21.593017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.593017    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc23b07-2906-4a12-b45a-7064e832aedc" containerName="revoke-keys"
Nov 02 21:28:21.593127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.593098    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc23b07-2906-4a12-b45a-7064e832aedc" containerName="revoke-keys"
Nov 02 21:28:21.596322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.596308    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.642098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.642058    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vh7\" (UniqueName: \"kubernetes.io/projected/449b2282-4642-479e-9fcb-b01db91a5697-kube-api-access-56vh7\") pod \"maas-api-revoke-sub-2044f278-6qhqf\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.642294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.642102    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2044f278-6qhqf\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.642294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.642214    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2044f278-6qhqf\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.649982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.649955    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"]
Nov 02 21:28:21.697323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.697290    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:28:21.697323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.697296    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:28:21.697529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.697329    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:28:21.698484 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.698466    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:28:21.743401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.743370    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vh7\" (UniqueName: \"kubernetes.io/projected/449b2282-4642-479e-9fcb-b01db91a5697-kube-api-access-56vh7\") pod \"maas-api-revoke-sub-2044f278-6qhqf\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.743401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.743400    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2044f278-6qhqf\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.743608 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.743418    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2044f278-6qhqf\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.744154 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.744135    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2044f278-6qhqf\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.745754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.745736    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2044f278-6qhqf\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.800408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.800380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vh7\" (UniqueName: \"kubernetes.io/projected/449b2282-4642-479e-9fcb-b01db91a5697-kube-api-access-56vh7\") pod \"maas-api-revoke-sub-2044f278-6qhqf\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:21.909091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:21.908991    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:22.276615 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:28:22.276586    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449b2282_4642_479e_9fcb_b01db91a5697.slice/crio-c09df8b252990a63bdb9930823bfb2b9c1be6e0fb262b7b9e0cd148c19986918 WatchSource:0}: Error finding container c09df8b252990a63bdb9930823bfb2b9c1be6e0fb262b7b9e0cd148c19986918: Status 404 returned error can't find the container with id c09df8b252990a63bdb9930823bfb2b9c1be6e0fb262b7b9e0cd148c19986918
Nov 02 21:28:22.279627 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:22.279602    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"]
Nov 02 21:28:22.301794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:22.301765    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf" event={"ID":"449b2282-4642-479e-9fcb-b01db91a5697","Type":"ContainerStarted","Data":"c09df8b252990a63bdb9930823bfb2b9c1be6e0fb262b7b9e0cd148c19986918"}
Nov 02 21:28:23.308578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:23.308527    4528 generic.go:356] "Generic (PLEG): container finished" podID="449b2282-4642-479e-9fcb-b01db91a5697" containerID="3e81694d3a2a3317e5fab489347a0ec198baeab9ca6642ade33b46a7bc5cad26" exitCode=0
Nov 02 21:28:23.309068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:23.308644    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf" event={"ID":"449b2282-4642-479e-9fcb-b01db91a5697","Type":"ContainerDied","Data":"3e81694d3a2a3317e5fab489347a0ec198baeab9ca6642ade33b46a7bc5cad26"}
Nov 02 21:28:24.454041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.454015    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:24.576459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.576383    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449b2282-4642-479e-9fcb-b01db91a5697-kube-api-access-56vh7\" (UniqueName: \"kubernetes.io/projected/449b2282-4642-479e-9fcb-b01db91a5697-kube-api-access-56vh7\") pod \"449b2282-4642-479e-9fcb-b01db91a5697\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") "
Nov 02 21:28:24.576619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.576487    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-xks-ca\") pod \"449b2282-4642-479e-9fcb-b01db91a5697\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") "
Nov 02 21:28:24.576619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.576550    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-openshift-ca\") pod \"449b2282-4642-479e-9fcb-b01db91a5697\" (UID: \"449b2282-4642-479e-9fcb-b01db91a5697\") "
Nov 02 21:28:24.576941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.576900    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-openshift-ca" pod "449b2282-4642-479e-9fcb-b01db91a5697" (UID: "449b2282-4642-479e-9fcb-b01db91a5697"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:28:24.577096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.577072    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:24.578538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.578514    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-xks-ca" pod "449b2282-4642-479e-9fcb-b01db91a5697" (UID: "449b2282-4642-479e-9fcb-b01db91a5697"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:28:24.578649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.578594    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449b2282-4642-479e-9fcb-b01db91a5697-kube-api-access-56vh7" pod "449b2282-4642-479e-9fcb-b01db91a5697" (UID: "449b2282-4642-479e-9fcb-b01db91a5697"). InnerVolumeSpecName "kube-api-access-56vh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:28:24.678289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.678246    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56vh7\" (UniqueName: \"kubernetes.io/projected/449b2282-4642-479e-9fcb-b01db91a5697-kube-api-access-56vh7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:24.678289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:24.678282    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449b2282-4642-479e-9fcb-b01db91a5697-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:25.321163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:25.321133    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"
Nov 02 21:28:25.321163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:25.321137    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf" event={"ID":"449b2282-4642-479e-9fcb-b01db91a5697","Type":"ContainerDied","Data":"c09df8b252990a63bdb9930823bfb2b9c1be6e0fb262b7b9e0cd148c19986918"}
Nov 02 21:28:25.321419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:25.321186    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09df8b252990a63bdb9930823bfb2b9c1be6e0fb262b7b9e0cd148c19986918"
Nov 02 21:28:28.397998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.397962    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"]
Nov 02 21:28:28.398459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.398443    4528 cpu_manager.go:414] "containerMap: removing container" podUID="449b2282-4642-479e-9fcb-b01db91a5697" containerName="revoke-keys"
Nov 02 21:28:28.398503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.398462    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449b2282-4642-479e-9fcb-b01db91a5697" containerName="revoke-keys"
Nov 02 21:28:28.398588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.398580    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="449b2282-4642-479e-9fcb-b01db91a5697" containerName="revoke-keys"
Nov 02 21:28:28.401833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.401816    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.454330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.454297    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"]
Nov 02 21:28:28.501420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.501387    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:28:28.501420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.501402    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:28:28.501622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.501459    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:28:28.501622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.501488    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:28:28.520048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.520025    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9345bd2a-z5qs4\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.520190 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.520055    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9345bd2a-z5qs4\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.520190 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.520174    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8rr\" (UniqueName: \"kubernetes.io/projected/c11666a4-2dd6-405d-b328-1560dd6812ac-kube-api-access-cb8rr\") pod \"maas-api-revoke-sub-9345bd2a-z5qs4\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.621468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.621430    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9345bd2a-z5qs4\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.621468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.621470    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9345bd2a-z5qs4\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.621710 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.621515    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8rr\" (UniqueName: \"kubernetes.io/projected/c11666a4-2dd6-405d-b328-1560dd6812ac-kube-api-access-cb8rr\") pod \"maas-api-revoke-sub-9345bd2a-z5qs4\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.622194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.622167    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9345bd2a-z5qs4\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.623740 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.623717    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9345bd2a-z5qs4\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.678282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.678232    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8rr\" (UniqueName: \"kubernetes.io/projected/c11666a4-2dd6-405d-b328-1560dd6812ac-kube-api-access-cb8rr\") pod \"maas-api-revoke-sub-9345bd2a-z5qs4\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:28.714084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:28.714059    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:29.078654 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:28:29.078624    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11666a4_2dd6_405d_b328_1560dd6812ac.slice/crio-0f284e25c7d4c020ddfce0a30b7c9a4bb78675d8e64305a907609efc52850373 WatchSource:0}: Error finding container 0f284e25c7d4c020ddfce0a30b7c9a4bb78675d8e64305a907609efc52850373: Status 404 returned error can't find the container with id 0f284e25c7d4c020ddfce0a30b7c9a4bb78675d8e64305a907609efc52850373
Nov 02 21:28:29.082924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:29.082899    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"]
Nov 02 21:28:29.345864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:29.345792    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4" event={"ID":"c11666a4-2dd6-405d-b328-1560dd6812ac","Type":"ContainerStarted","Data":"0f284e25c7d4c020ddfce0a30b7c9a4bb78675d8e64305a907609efc52850373"}
Nov 02 21:28:30.352763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:30.352726    4528 generic.go:356] "Generic (PLEG): container finished" podID="c11666a4-2dd6-405d-b328-1560dd6812ac" containerID="c50391be9baf6ffe6720eadfd2a9cd9222976cecb1542401871a9a73538be1a7" exitCode=0
Nov 02 21:28:30.353186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:30.352812    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4" event={"ID":"c11666a4-2dd6-405d-b328-1560dd6812ac","Type":"ContainerDied","Data":"c50391be9baf6ffe6720eadfd2a9cd9222976cecb1542401871a9a73538be1a7"}
Nov 02 21:28:31.502047 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.502023    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:31.656397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.656322    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-xks-ca\") pod \"c11666a4-2dd6-405d-b328-1560dd6812ac\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") "
Nov 02 21:28:31.656543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.656523    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-openshift-ca\") pod \"c11666a4-2dd6-405d-b328-1560dd6812ac\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") "
Nov 02 21:28:31.656590 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.656544    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11666a4-2dd6-405d-b328-1560dd6812ac-kube-api-access-cb8rr\" (UniqueName: \"kubernetes.io/projected/c11666a4-2dd6-405d-b328-1560dd6812ac-kube-api-access-cb8rr\") pod \"c11666a4-2dd6-405d-b328-1560dd6812ac\" (UID: \"c11666a4-2dd6-405d-b328-1560dd6812ac\") "
Nov 02 21:28:31.656922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.656894    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-openshift-ca" pod "c11666a4-2dd6-405d-b328-1560dd6812ac" (UID: "c11666a4-2dd6-405d-b328-1560dd6812ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:28:31.658442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.658417    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-xks-ca" pod "c11666a4-2dd6-405d-b328-1560dd6812ac" (UID: "c11666a4-2dd6-405d-b328-1560dd6812ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:28:31.658683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.658663    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11666a4-2dd6-405d-b328-1560dd6812ac-kube-api-access-cb8rr" pod "c11666a4-2dd6-405d-b328-1560dd6812ac" (UID: "c11666a4-2dd6-405d-b328-1560dd6812ac"). InnerVolumeSpecName "kube-api-access-cb8rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:28:31.757852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.757819    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:31.757852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.757847    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8rr\" (UniqueName: \"kubernetes.io/projected/c11666a4-2dd6-405d-b328-1560dd6812ac-kube-api-access-cb8rr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:31.757852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:31.757858    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11666a4-2dd6-405d-b328-1560dd6812ac-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:32.339066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.339029    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"]
Nov 02 21:28:32.339548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.339535    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c11666a4-2dd6-405d-b328-1560dd6812ac" containerName="revoke-keys"
Nov 02 21:28:32.339611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.339550    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11666a4-2dd6-405d-b328-1560dd6812ac" containerName="revoke-keys"
Nov 02 21:28:32.339649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.339641    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11666a4-2dd6-405d-b328-1560dd6812ac" containerName="revoke-keys"
Nov 02 21:28:32.342863 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.342846    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.364822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.364796    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"
Nov 02 21:28:32.365000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.364796    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4" event={"ID":"c11666a4-2dd6-405d-b328-1560dd6812ac","Type":"ContainerDied","Data":"0f284e25c7d4c020ddfce0a30b7c9a4bb78675d8e64305a907609efc52850373"}
Nov 02 21:28:32.365000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.364911    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f284e25c7d4c020ddfce0a30b7c9a4bb78675d8e64305a907609efc52850373"
Nov 02 21:28:32.396534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.396506    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"]
Nov 02 21:28:32.464877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.464845    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a61fd519-5766n\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.465085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.465019    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a61fd519-5766n\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.465085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.465080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hwz\" (UniqueName: \"kubernetes.io/projected/011aac80-b631-471d-8224-b0303bf9b6ff-kube-api-access-82hwz\") pod \"maas-api-revoke-sub-a61fd519-5766n\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.566688 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.566655    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a61fd519-5766n\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.567042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.566730    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a61fd519-5766n\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.567042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.566778    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hwz\" (UniqueName: \"kubernetes.io/projected/011aac80-b631-471d-8224-b0303bf9b6ff-kube-api-access-82hwz\") pod \"maas-api-revoke-sub-a61fd519-5766n\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.567303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.567284    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a61fd519-5766n\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.568912 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.568892    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a61fd519-5766n\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.623757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.623684    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hwz\" (UniqueName: \"kubernetes.io/projected/011aac80-b631-471d-8224-b0303bf9b6ff-kube-api-access-82hwz\") pod \"maas-api-revoke-sub-a61fd519-5766n\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:32.656731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:32.656701    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:33.025673 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:33.025641    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"]
Nov 02 21:28:33.030058 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:28:33.030025    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011aac80_b631_471d_8224_b0303bf9b6ff.slice/crio-7c39bcf51f4f9bbf599ee14f56e026c8ef4ff693b2d67f8dffce065115e77f05 WatchSource:0}: Error finding container 7c39bcf51f4f9bbf599ee14f56e026c8ef4ff693b2d67f8dffce065115e77f05: Status 404 returned error can't find the container with id 7c39bcf51f4f9bbf599ee14f56e026c8ef4ff693b2d67f8dffce065115e77f05
Nov 02 21:28:33.371857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:33.371826    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n" event={"ID":"011aac80-b631-471d-8224-b0303bf9b6ff","Type":"ContainerStarted","Data":"4efddffeb0c5fe98e5b9528174fc7bdd6ea08e5d7d5464f487c6777f2229ed64"}
Nov 02 21:28:33.371988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:33.371877    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n" event={"ID":"011aac80-b631-471d-8224-b0303bf9b6ff","Type":"ContainerStarted","Data":"7c39bcf51f4f9bbf599ee14f56e026c8ef4ff693b2d67f8dffce065115e77f05"}
Nov 02 21:28:33.485456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:33.485408    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n" podStartSLOduration=1.485394406 podStartE2EDuration="1.485394406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:33.48453654 +0000 UTC m=+1200.748072857" watchObservedRunningTime="2026-11-02 21:28:33.485394406 +0000 UTC m=+1200.748930753"
Nov 02 21:28:34.379355 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:34.379324    4528 generic.go:356] "Generic (PLEG): container finished" podID="011aac80-b631-471d-8224-b0303bf9b6ff" containerID="4efddffeb0c5fe98e5b9528174fc7bdd6ea08e5d7d5464f487c6777f2229ed64" exitCode=0
Nov 02 21:28:34.379758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:34.379364    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n" event={"ID":"011aac80-b631-471d-8224-b0303bf9b6ff","Type":"ContainerDied","Data":"4efddffeb0c5fe98e5b9528174fc7bdd6ea08e5d7d5464f487c6777f2229ed64"}
Nov 02 21:28:35.857265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:35.850520    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:35.945204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:35.945176    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011aac80-b631-471d-8224-b0303bf9b6ff-kube-api-access-82hwz\" (UniqueName: \"kubernetes.io/projected/011aac80-b631-471d-8224-b0303bf9b6ff-kube-api-access-82hwz\") pod \"011aac80-b631-471d-8224-b0303bf9b6ff\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") "
Nov 02 21:28:35.945407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:35.945253    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-openshift-ca\") pod \"011aac80-b631-471d-8224-b0303bf9b6ff\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") "
Nov 02 21:28:35.945407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:35.945291    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-xks-ca\") pod \"011aac80-b631-471d-8224-b0303bf9b6ff\" (UID: \"011aac80-b631-471d-8224-b0303bf9b6ff\") "
Nov 02 21:28:35.945729 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:35.945696    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-openshift-ca" pod "011aac80-b631-471d-8224-b0303bf9b6ff" (UID: "011aac80-b631-471d-8224-b0303bf9b6ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:28:35.947891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:35.947849    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-xks-ca" pod "011aac80-b631-471d-8224-b0303bf9b6ff" (UID: "011aac80-b631-471d-8224-b0303bf9b6ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:28:35.948012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:35.947951    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011aac80-b631-471d-8224-b0303bf9b6ff-kube-api-access-82hwz" pod "011aac80-b631-471d-8224-b0303bf9b6ff" (UID: "011aac80-b631-471d-8224-b0303bf9b6ff"). InnerVolumeSpecName "kube-api-access-82hwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:28:36.046773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:36.046690    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hwz\" (UniqueName: \"kubernetes.io/projected/011aac80-b631-471d-8224-b0303bf9b6ff-kube-api-access-82hwz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:36.046773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:36.046716    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:36.046773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:36.046726    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011aac80-b631-471d-8224-b0303bf9b6ff-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:36.394213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:36.394105    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"
Nov 02 21:28:36.394213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:36.394154    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n" event={"ID":"011aac80-b631-471d-8224-b0303bf9b6ff","Type":"ContainerDied","Data":"7c39bcf51f4f9bbf599ee14f56e026c8ef4ff693b2d67f8dffce065115e77f05"}
Nov 02 21:28:36.394213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:36.394205    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c39bcf51f4f9bbf599ee14f56e026c8ef4ff693b2d67f8dffce065115e77f05"
Nov 02 21:28:43.816797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.816763    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"]
Nov 02 21:28:43.817294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.817279    4528 cpu_manager.go:414] "containerMap: removing container" podUID="011aac80-b631-471d-8224-b0303bf9b6ff" containerName="revoke-keys"
Nov 02 21:28:43.817338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.817295    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011aac80-b631-471d-8224-b0303bf9b6ff" containerName="revoke-keys"
Nov 02 21:28:43.817378 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.817370    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="011aac80-b631-471d-8224-b0303bf9b6ff" containerName="revoke-keys"
Nov 02 21:28:43.820826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.820810    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:43.872920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.872877    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"]
Nov 02 21:28:43.919582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.919553    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:28:43.919582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.919553    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:28:43.919582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.919553    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:28:43.923249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.923235    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:28:43.928212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.928176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef178d2-zmnrs\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:43.928278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.928222    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whkvv\" (UniqueName: \"kubernetes.io/projected/916bdd95-3ab7-47b9-a385-c545e5de6a55-kube-api-access-whkvv\") pod \"maas-api-revoke-sub-4ef178d2-zmnrs\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:43.928325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:43.928313    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef178d2-zmnrs\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:44.029494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:44.029467    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef178d2-zmnrs\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:44.029662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:44.029503    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkvv\" (UniqueName: \"kubernetes.io/projected/916bdd95-3ab7-47b9-a385-c545e5de6a55-kube-api-access-whkvv\") pod \"maas-api-revoke-sub-4ef178d2-zmnrs\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:44.029662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:44.029543    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef178d2-zmnrs\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:44.030221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:44.030191    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef178d2-zmnrs\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:44.031757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:44.031736    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef178d2-zmnrs\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:44.086950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:44.086882    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkvv\" (UniqueName: \"kubernetes.io/projected/916bdd95-3ab7-47b9-a385-c545e5de6a55-kube-api-access-whkvv\") pod \"maas-api-revoke-sub-4ef178d2-zmnrs\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:44.112722 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:28:44.112685    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:28:44.133872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:44.133846    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:44.501175 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:28:44.501105    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916bdd95_3ab7_47b9_a385_c545e5de6a55.slice/crio-8053cccbdc7cc03d2a8b74d36f310deca9dc9e25d4b59bc5bd7102afdbd3cb5f WatchSource:0}: Error finding container 8053cccbdc7cc03d2a8b74d36f310deca9dc9e25d4b59bc5bd7102afdbd3cb5f: Status 404 returned error can't find the container with id 8053cccbdc7cc03d2a8b74d36f310deca9dc9e25d4b59bc5bd7102afdbd3cb5f
Nov 02 21:28:44.503260 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:44.503234    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"]
Nov 02 21:28:45.459244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:45.459203    4528 generic.go:356] "Generic (PLEG): container finished" podID="916bdd95-3ab7-47b9-a385-c545e5de6a55" containerID="4928d24e4b1b57bfbc3500fd3cf889f1614109069ee004c1af68dbf14ff1d3e4" exitCode=0
Nov 02 21:28:45.459725 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:45.459400    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs" event={"ID":"916bdd95-3ab7-47b9-a385-c545e5de6a55","Type":"ContainerDied","Data":"4928d24e4b1b57bfbc3500fd3cf889f1614109069ee004c1af68dbf14ff1d3e4"}
Nov 02 21:28:45.459725 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:45.459437    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs" event={"ID":"916bdd95-3ab7-47b9-a385-c545e5de6a55","Type":"ContainerStarted","Data":"8053cccbdc7cc03d2a8b74d36f310deca9dc9e25d4b59bc5bd7102afdbd3cb5f"}
Nov 02 21:28:46.597008 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.596963    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:46.762462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.762431    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916bdd95-3ab7-47b9-a385-c545e5de6a55-kube-api-access-whkvv\" (UniqueName: \"kubernetes.io/projected/916bdd95-3ab7-47b9-a385-c545e5de6a55-kube-api-access-whkvv\") pod \"916bdd95-3ab7-47b9-a385-c545e5de6a55\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") "
Nov 02 21:28:46.762609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.762563    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-xks-ca\") pod \"916bdd95-3ab7-47b9-a385-c545e5de6a55\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") "
Nov 02 21:28:46.762609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.762599    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-openshift-ca\") pod \"916bdd95-3ab7-47b9-a385-c545e5de6a55\" (UID: \"916bdd95-3ab7-47b9-a385-c545e5de6a55\") "
Nov 02 21:28:46.762997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.762970    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-openshift-ca" pod "916bdd95-3ab7-47b9-a385-c545e5de6a55" (UID: "916bdd95-3ab7-47b9-a385-c545e5de6a55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:28:46.764519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.764490    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916bdd95-3ab7-47b9-a385-c545e5de6a55-kube-api-access-whkvv" pod "916bdd95-3ab7-47b9-a385-c545e5de6a55" (UID: "916bdd95-3ab7-47b9-a385-c545e5de6a55"). InnerVolumeSpecName "kube-api-access-whkvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:28:46.764613 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.764553    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-xks-ca" pod "916bdd95-3ab7-47b9-a385-c545e5de6a55" (UID: "916bdd95-3ab7-47b9-a385-c545e5de6a55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:28:46.864152 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.864092    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whkvv\" (UniqueName: \"kubernetes.io/projected/916bdd95-3ab7-47b9-a385-c545e5de6a55-kube-api-access-whkvv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:46.864152 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.864148    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:46.864152 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:46.864159    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916bdd95-3ab7-47b9-a385-c545e5de6a55-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:47.471283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:47.471256    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"
Nov 02 21:28:47.471454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:47.471282    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs" event={"ID":"916bdd95-3ab7-47b9-a385-c545e5de6a55","Type":"ContainerDied","Data":"8053cccbdc7cc03d2a8b74d36f310deca9dc9e25d4b59bc5bd7102afdbd3cb5f"}
Nov 02 21:28:47.471454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:47.471328    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8053cccbdc7cc03d2a8b74d36f310deca9dc9e25d4b59bc5bd7102afdbd3cb5f"
Nov 02 21:28:48.692660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.692625    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"]
Nov 02 21:28:48.693181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.693166    4528 cpu_manager.go:414] "containerMap: removing container" podUID="916bdd95-3ab7-47b9-a385-c545e5de6a55" containerName="revoke-keys"
Nov 02 21:28:48.693226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.693182    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916bdd95-3ab7-47b9-a385-c545e5de6a55" containerName="revoke-keys"
Nov 02 21:28:48.693276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.693265    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="916bdd95-3ab7-47b9-a385-c545e5de6a55" containerName="revoke-keys"
Nov 02 21:28:48.696646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.696632    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:48.751572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.751545    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"]
Nov 02 21:28:48.784662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.784630    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6391aeaa-pgvsj\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:48.784804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.784709    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6391aeaa-pgvsj\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:48.784804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.784772    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhz6f\" (UniqueName: \"kubernetes.io/projected/fb2f078e-f6ac-4026-80c6-2dea29cc2294-kube-api-access-nhz6f\") pod \"maas-api-revoke-sub-6391aeaa-pgvsj\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:48.798686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.798659    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:28:48.798686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.798679    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:28:48.798841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.798741    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:28:48.800535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.800523    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:28:48.886349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.886318    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6391aeaa-pgvsj\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:48.886494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.886404    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6391aeaa-pgvsj\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:48.886494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.886432    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhz6f\" (UniqueName: \"kubernetes.io/projected/fb2f078e-f6ac-4026-80c6-2dea29cc2294-kube-api-access-nhz6f\") pod \"maas-api-revoke-sub-6391aeaa-pgvsj\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:48.886966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.886941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6391aeaa-pgvsj\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:48.888659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.888641    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6391aeaa-pgvsj\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:48.943257 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:48.943191    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhz6f\" (UniqueName: \"kubernetes.io/projected/fb2f078e-f6ac-4026-80c6-2dea29cc2294-kube-api-access-nhz6f\") pod \"maas-api-revoke-sub-6391aeaa-pgvsj\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:49.009165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:49.009136    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:49.378101 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:28:49.378068    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2f078e_f6ac_4026_80c6_2dea29cc2294.slice/crio-f84e8e8bfe9b63924ca65166740fb32a474ba4b8f04589822fbb86e7a862532e WatchSource:0}: Error finding container f84e8e8bfe9b63924ca65166740fb32a474ba4b8f04589822fbb86e7a862532e: Status 404 returned error can't find the container with id f84e8e8bfe9b63924ca65166740fb32a474ba4b8f04589822fbb86e7a862532e
Nov 02 21:28:49.381076 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:49.381039    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"]
Nov 02 21:28:49.484228 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:49.484197    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj" event={"ID":"fb2f078e-f6ac-4026-80c6-2dea29cc2294","Type":"ContainerStarted","Data":"f84e8e8bfe9b63924ca65166740fb32a474ba4b8f04589822fbb86e7a862532e"}
Nov 02 21:28:50.491356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:50.491322    4528 generic.go:356] "Generic (PLEG): container finished" podID="fb2f078e-f6ac-4026-80c6-2dea29cc2294" containerID="ec1b5342772bb18d436d546bee61b698281b20cc9c16b119956abcc1f03f2e69" exitCode=0
Nov 02 21:28:50.491796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:50.491410    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj" event={"ID":"fb2f078e-f6ac-4026-80c6-2dea29cc2294","Type":"ContainerDied","Data":"ec1b5342772bb18d436d546bee61b698281b20cc9c16b119956abcc1f03f2e69"}
Nov 02 21:28:51.647917 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.647892    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:51.816701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.816666    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2f078e-f6ac-4026-80c6-2dea29cc2294-kube-api-access-nhz6f\" (UniqueName: \"kubernetes.io/projected/fb2f078e-f6ac-4026-80c6-2dea29cc2294-kube-api-access-nhz6f\") pod \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") "
Nov 02 21:28:51.816881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.816769    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-xks-ca\") pod \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") "
Nov 02 21:28:51.816881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.816807    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-openshift-ca\") pod \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\" (UID: \"fb2f078e-f6ac-4026-80c6-2dea29cc2294\") "
Nov 02 21:28:51.817243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.817218    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-openshift-ca" pod "fb2f078e-f6ac-4026-80c6-2dea29cc2294" (UID: "fb2f078e-f6ac-4026-80c6-2dea29cc2294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:28:51.818720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.818701    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-xks-ca" pod "fb2f078e-f6ac-4026-80c6-2dea29cc2294" (UID: "fb2f078e-f6ac-4026-80c6-2dea29cc2294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:28:51.818803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.818761    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2f078e-f6ac-4026-80c6-2dea29cc2294-kube-api-access-nhz6f" pod "fb2f078e-f6ac-4026-80c6-2dea29cc2294" (UID: "fb2f078e-f6ac-4026-80c6-2dea29cc2294"). InnerVolumeSpecName "kube-api-access-nhz6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:28:51.917801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.917773    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:51.917801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.917801    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f078e-f6ac-4026-80c6-2dea29cc2294-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:51.917966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:51.917810    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhz6f\" (UniqueName: \"kubernetes.io/projected/fb2f078e-f6ac-4026-80c6-2dea29cc2294-kube-api-access-nhz6f\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:28:52.503648 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.503621    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"
Nov 02 21:28:52.503820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.503652    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj" event={"ID":"fb2f078e-f6ac-4026-80c6-2dea29cc2294","Type":"ContainerDied","Data":"f84e8e8bfe9b63924ca65166740fb32a474ba4b8f04589822fbb86e7a862532e"}
Nov 02 21:28:52.503820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.503694    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f84e8e8bfe9b63924ca65166740fb32a474ba4b8f04589822fbb86e7a862532e"
Nov 02 21:28:52.761950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.761918    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"]
Nov 02 21:28:52.762499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.762482    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fb2f078e-f6ac-4026-80c6-2dea29cc2294" containerName="revoke-keys"
Nov 02 21:28:52.762499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.762499    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2f078e-f6ac-4026-80c6-2dea29cc2294" containerName="revoke-keys"
Nov 02 21:28:52.762640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.762574    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2f078e-f6ac-4026-80c6-2dea29cc2294" containerName="revoke-keys"
Nov 02 21:28:52.765459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.765442    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:52.817881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.817848    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"]
Nov 02 21:28:52.831511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.831483    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"distinct-2-w0-0c9014-kserve-self-signed-certs\""
Nov 02 21:28:52.930624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.930588    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-dshm\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:52.931171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.930984    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45n5n\" (UniqueName: \"kubernetes.io/projected/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-kube-api-access-45n5n\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:52.931408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.931385    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tls-certs\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:52.933024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.932677    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tmp-dir\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:52.933024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.932788    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-model-cache\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:52.933024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:52.932851    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-home\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.034323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.034235    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-model-cache\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.034486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.034386    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-home\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.034486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.034437    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-dshm\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.034610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.034502    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45n5n\" (UniqueName: \"kubernetes.io/projected/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-kube-api-access-45n5n\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.034610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.034545    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tls-certs\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.034610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.034598    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tmp-dir\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.034757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.034649    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-model-cache\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.034757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.034686    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-home\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.034955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.034928    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tmp-dir\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.036925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.036888    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-dshm\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.037175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.037156    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tls-certs\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.091414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.091388    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45n5n\" (UniqueName: \"kubernetes.io/projected/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-kube-api-access-45n5n\") pod \"distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") " pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.383980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.383890    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:53.750947 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:28:53.750848    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e4e9fc7_34d3_4d98_87f5_9faf1efd4f0d.slice/crio-cf344fc56d9372554e81c70f177ac4f46b1e0f0422c0efe2f3a6f9deb5029461 WatchSource:0}: Error finding container cf344fc56d9372554e81c70f177ac4f46b1e0f0422c0efe2f3a6f9deb5029461: Status 404 returned error can't find the container with id cf344fc56d9372554e81c70f177ac4f46b1e0f0422c0efe2f3a6f9deb5029461
Nov 02 21:28:53.754281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:53.754257    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"]
Nov 02 21:28:54.516794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:54.516759    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" event={"ID":"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d","Type":"ContainerStarted","Data":"143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449"}
Nov 02 21:28:54.517289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:54.516805    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" event={"ID":"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d","Type":"ContainerStarted","Data":"cf344fc56d9372554e81c70f177ac4f46b1e0f0422c0efe2f3a6f9deb5029461"}
Nov 02 21:28:54.633790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:54.633738    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podStartSLOduration=2.633724649 podStartE2EDuration="2.633724649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:54.631612621 +0000 UTC m=+1221.895148940" watchObservedRunningTime="2026-11-02 21:28:54.633724649 +0000 UTC m=+1221.897260967"
Nov 02 21:28:55.263474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:55.263436    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:28:55.384019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:55.383984    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:28:56.276793 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.276758    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"]
Nov 02 21:28:56.282821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.282796    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.445134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.407055    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.490307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.490265    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhwr4\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-kube-api-access-fhwr4\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.490565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.490545    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.490707 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.490689    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.490843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.490827    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.491083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.491065    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.491283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.491263    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.491416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.491401    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.491526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.491513    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.491759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.447141    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"]
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.602753    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhwr4\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-kube-api-access-fhwr4\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.602811    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.602842    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.602898    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.603011    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.603046    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.603081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.603121    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.603211    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.606629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.606899    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.607167    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.618195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.607934    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.652132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.644893    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.674344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.674313    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.687086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.687055    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.702032 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.701998    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.726177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.715790    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhwr4\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-kube-api-access-fhwr4\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:56.921289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:56.921192    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:57.742463 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:28:57.742430    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded01ac5b_9b7a_42b8_8053_d2e0c53e149a.slice/crio-794c0f7213060b6f8b77ba556988f781665b47943e4a4b66df8b2521c5ce6c5b WatchSource:0}: Error finding container 794c0f7213060b6f8b77ba556988f781665b47943e4a4b66df8b2521c5ce6c5b: Status 404 returned error can't find the container with id 794c0f7213060b6f8b77ba556988f781665b47943e4a4b66df8b2521c5ce6c5b
Nov 02 21:28:57.744913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:57.744880    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:28:57.745039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:57.744954    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:28:57.745039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:57.744999    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:28:57.745563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:57.745541    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"]
Nov 02 21:28:58.690644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:58.690601    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" event={"ID":"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a","Type":"ContainerStarted","Data":"008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8"}
Nov 02 21:28:58.690825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:58.690668    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" event={"ID":"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a","Type":"ContainerStarted","Data":"794c0f7213060b6f8b77ba556988f781665b47943e4a4b66df8b2521c5ce6c5b"}
Nov 02 21:28:58.924133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:58.923507    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" podStartSLOduration=2.923486305 podStartE2EDuration="2.923486305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:58.856884102 +0000 UTC m=+1226.120420418" watchObservedRunningTime="2026-11-02 21:28:58.923486305 +0000 UTC m=+1226.187022623"
Nov 02 21:28:58.929729 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:58.929697    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 21:28:58.929915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:58.929753    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 21:28:58.935558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:58.935532    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:28:59.924133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:59.923497    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 21:28:59.924133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:59.923582    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 21:28:59.980560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:59.980526    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"]
Nov 02 21:28:59.984519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:28:59.984503    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.038036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.038002    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"]
Nov 02 21:29:00.041814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.041782    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbe882e8-4w884\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.041961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.041936    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbe882e8-4w884\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.042047 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.041974    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq2tr\" (UniqueName: \"kubernetes.io/projected/e77218fa-47f1-4a29-9efb-420c1c0df91e-kube-api-access-qq2tr\") pod \"maas-api-revoke-sub-bbe882e8-4w884\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.083185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.083162    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:29:00.083302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.083222    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:29:00.087640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.087625    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:29:00.087691 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.087647    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:29:00.142768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.142738    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbe882e8-4w884\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.142929 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.142824    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbe882e8-4w884\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.142929 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.142864    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2tr\" (UniqueName: \"kubernetes.io/projected/e77218fa-47f1-4a29-9efb-420c1c0df91e-kube-api-access-qq2tr\") pod \"maas-api-revoke-sub-bbe882e8-4w884\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.143699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.143677    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbe882e8-4w884\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.145246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.145219    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbe882e8-4w884\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.200844 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.200784    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2tr\" (UniqueName: \"kubernetes.io/projected/e77218fa-47f1-4a29-9efb-420c1c0df91e-kube-api-access-qq2tr\") pod \"maas-api-revoke-sub-bbe882e8-4w884\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.297558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.297531    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:00.691363 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:00.691333    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77218fa_47f1_4a29_9efb_420c1c0df91e.slice/crio-d65f6c6bdd338b4debec4772e41398d422e1e197e7c8440851386d4079162b56 WatchSource:0}: Error finding container d65f6c6bdd338b4debec4772e41398d422e1e197e7c8440851386d4079162b56: Status 404 returned error can't find the container with id d65f6c6bdd338b4debec4772e41398d422e1e197e7c8440851386d4079162b56
Nov 02 21:29:00.694178 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.694150    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"]
Nov 02 21:29:00.719719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.719684    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884" event={"ID":"e77218fa-47f1-4a29-9efb-420c1c0df91e","Type":"ContainerStarted","Data":"d65f6c6bdd338b4debec4772e41398d422e1e197e7c8440851386d4079162b56"}
Nov 02 21:29:00.926472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.926341    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 21:29:00.926472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:00.926397    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 21:29:01.731020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:01.730982    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884" event={"ID":"e77218fa-47f1-4a29-9efb-420c1c0df91e","Type":"ContainerStarted","Data":"4689780761d377e86d4d857dbe93c539c0bd2f3f94d6e15d8f9dbf6bbcedbab9"}
Nov 02 21:29:01.845760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:01.845706    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884" podStartSLOduration=2.845690399 podStartE2EDuration="2.845690399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:01.84389966 +0000 UTC m=+1229.107435977" watchObservedRunningTime="2026-11-02 21:29:01.845690399 +0000 UTC m=+1229.109226717"
Nov 02 21:29:01.925234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:01.925196    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 21:29:01.925501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:01.925470    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 21:29:02.740057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:02.740007    4528 generic.go:356] "Generic (PLEG): container finished" podID="e77218fa-47f1-4a29-9efb-420c1c0df91e" containerID="4689780761d377e86d4d857dbe93c539c0bd2f3f94d6e15d8f9dbf6bbcedbab9" exitCode=0
Nov 02 21:29:02.740567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:02.740102    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884" event={"ID":"e77218fa-47f1-4a29-9efb-420c1c0df91e","Type":"ContainerDied","Data":"4689780761d377e86d4d857dbe93c539c0bd2f3f94d6e15d8f9dbf6bbcedbab9"}
Nov 02 21:29:02.927936 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:02.926923    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 21:29:02.927936 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:02.926984    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 21:29:03.056321 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.056290    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"]
Nov 02 21:29:03.059163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.059141    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041927d0-ffttj"]
Nov 02 21:29:03.384819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.384741    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:29:03.397329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.397305    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:29:03.765458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.765419    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:29:03.939173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.939095    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 02 21:29:03.939418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.939388    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 02 21:29:03.946806 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.946763    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:03.987093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.987059    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-openshift-ca\") pod \"e77218fa-47f1-4a29-9efb-420c1c0df91e\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") "
Nov 02 21:29:03.987281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.987105    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-xks-ca\") pod \"e77218fa-47f1-4a29-9efb-420c1c0df91e\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") "
Nov 02 21:29:03.987281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.987246    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77218fa-47f1-4a29-9efb-420c1c0df91e-kube-api-access-qq2tr\" (UniqueName: \"kubernetes.io/projected/e77218fa-47f1-4a29-9efb-420c1c0df91e-kube-api-access-qq2tr\") pod \"e77218fa-47f1-4a29-9efb-420c1c0df91e\" (UID: \"e77218fa-47f1-4a29-9efb-420c1c0df91e\") "
Nov 02 21:29:03.987475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.987452    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-openshift-ca" pod "e77218fa-47f1-4a29-9efb-420c1c0df91e" (UID: "e77218fa-47f1-4a29-9efb-420c1c0df91e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:29:03.987885 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.987862    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:03.989167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.989147    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-xks-ca" pod "e77218fa-47f1-4a29-9efb-420c1c0df91e" (UID: "e77218fa-47f1-4a29-9efb-420c1c0df91e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:29:03.989272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:03.989257    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77218fa-47f1-4a29-9efb-420c1c0df91e-kube-api-access-qq2tr" pod "e77218fa-47f1-4a29-9efb-420c1c0df91e" (UID: "e77218fa-47f1-4a29-9efb-420c1c0df91e"). InnerVolumeSpecName "kube-api-access-qq2tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:29:04.089679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:04.089636    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77218fa-47f1-4a29-9efb-420c1c0df91e-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:04.089861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:04.089679    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq2tr\" (UniqueName: \"kubernetes.io/projected/e77218fa-47f1-4a29-9efb-420c1c0df91e-kube-api-access-qq2tr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:04.757789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:04.757281    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"
Nov 02 21:29:04.757789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:04.757317    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884" event={"ID":"e77218fa-47f1-4a29-9efb-420c1c0df91e","Type":"ContainerDied","Data":"d65f6c6bdd338b4debec4772e41398d422e1e197e7c8440851386d4079162b56"}
Nov 02 21:29:04.757789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:04.757374    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d65f6c6bdd338b4debec4772e41398d422e1e197e7c8440851386d4079162b56"
Nov 02 21:29:04.912964 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:04.912918    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758b81bd-31a9-410e-868d-72ed8cdf9aa6" path="/var/lib/kubelet/pods/758b81bd-31a9-410e-868d-72ed8cdf9aa6/volumes"
Nov 02 21:29:05.346086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.346044    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:29:05.347055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.347029    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:29:05.348547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.348513    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 21:29:05.355669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.355644    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"]
Nov 02 21:29:05.356197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.356177    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e77218fa-47f1-4a29-9efb-420c1c0df91e" containerName="revoke-keys"
Nov 02 21:29:05.356197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.356196    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77218fa-47f1-4a29-9efb-420c1c0df91e" containerName="revoke-keys"
Nov 02 21:29:05.356366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.356317    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77218fa-47f1-4a29-9efb-420c1c0df91e" containerName="revoke-keys"
Nov 02 21:29:05.359986 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.359969    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.406917 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.406874    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxr7c\" (UniqueName: \"kubernetes.io/projected/0db59701-54b0-4f0c-b534-eeedcf9dc520-kube-api-access-lxr7c\") pod \"maas-api-revoke-sub-3bba4f0b-khx8r\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.407096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.407073    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bba4f0b-khx8r\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.407214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.407192    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bba4f0b-khx8r\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.449748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.449716    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:29:05.449748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.449716    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:29:05.449962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.449807    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:29:05.449962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.449810    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:29:05.452705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.452686    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"]
Nov 02 21:29:05.508208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.508178    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bba4f0b-khx8r\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.508369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.508272    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxr7c\" (UniqueName: \"kubernetes.io/projected/0db59701-54b0-4f0c-b534-eeedcf9dc520-kube-api-access-lxr7c\") pod \"maas-api-revoke-sub-3bba4f0b-khx8r\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.508369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.508335    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bba4f0b-khx8r\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.508868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.508842    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bba4f0b-khx8r\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.510704 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.510679    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bba4f0b-khx8r\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.565690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.565665    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxr7c\" (UniqueName: \"kubernetes.io/projected/0db59701-54b0-4f0c-b534-eeedcf9dc520-kube-api-access-lxr7c\") pod \"maas-api-revoke-sub-3bba4f0b-khx8r\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:05.673237 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:05.673169    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:06.043404 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:06.043371    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db59701_54b0_4f0c_b534_eeedcf9dc520.slice/crio-8996ca7e0eba7a350d8b3aa1ef7d83052127deb9b983a46d68678c8ff73fffcc WatchSource:0}: Error finding container 8996ca7e0eba7a350d8b3aa1ef7d83052127deb9b983a46d68678c8ff73fffcc: Status 404 returned error can't find the container with id 8996ca7e0eba7a350d8b3aa1ef7d83052127deb9b983a46d68678c8ff73fffcc
Nov 02 21:29:06.046346 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:06.046325    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"]
Nov 02 21:29:06.779174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:06.779137    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r" event={"ID":"0db59701-54b0-4f0c-b534-eeedcf9dc520","Type":"ContainerStarted","Data":"a9fa6fcb170fc12d224090f51d8c603debdcd012a1f1924473e55138aac6dfa4"}
Nov 02 21:29:06.779376 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:06.779196    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r" event={"ID":"0db59701-54b0-4f0c-b534-eeedcf9dc520","Type":"ContainerStarted","Data":"8996ca7e0eba7a350d8b3aa1ef7d83052127deb9b983a46d68678c8ff73fffcc"}
Nov 02 21:29:06.893417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:06.893365    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r" podStartSLOduration=1.8933504810000001 podStartE2EDuration="1.893350481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:06.892283138 +0000 UTC m=+1234.155819456" watchObservedRunningTime="2026-11-02 21:29:06.893350481 +0000 UTC m=+1234.156886799"
Nov 02 21:29:07.787884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:07.787847    4528 generic.go:356] "Generic (PLEG): container finished" podID="0db59701-54b0-4f0c-b534-eeedcf9dc520" containerID="a9fa6fcb170fc12d224090f51d8c603debdcd012a1f1924473e55138aac6dfa4" exitCode=0
Nov 02 21:29:07.788276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:07.787929    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r" event={"ID":"0db59701-54b0-4f0c-b534-eeedcf9dc520","Type":"ContainerDied","Data":"a9fa6fcb170fc12d224090f51d8c603debdcd012a1f1924473e55138aac6dfa4"}
Nov 02 21:29:08.194905 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.194828    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"]
Nov 02 21:29:08.198916 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.198895    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.238629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.238600    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.238762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.238643    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.238762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.238745    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l2l9\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-kube-api-access-7l2l9\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.238832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.238786    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.238866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.238836    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.238866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.238853    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.238928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.238888    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.238928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.238906    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47468c66-6cd7-4d1e-a705-0689ed95559a-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.239017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.238993    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340157    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340204    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340257    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l2l9\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-kube-api-access-7l2l9\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340276    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340314    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340336    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340384    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340408    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47468c66-6cd7-4d1e-a705-0689ed95559a-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340444    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340630    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340800    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.340893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.340854    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.341020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.341002    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.341223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.341200    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47468c66-6cd7-4d1e-a705-0689ed95559a-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.342604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.342582    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.342833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.342818    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.391175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.391142    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"]
Nov 02 21:29:08.397172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.397141    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.397461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.397437    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l2l9\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-kube-api-access-7l2l9\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.512010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.511980    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:08.935360 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.935334    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"]
Nov 02 21:29:08.936510 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.936492    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:08.965299 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:08.965267    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47468c66_6cd7_4d1e_a705_0689ed95559a.slice/crio-3733a69e91dcea54fba6b95acbf7430687c3103cefadc5b8f0ce56a5423b3b15 WatchSource:0}: Error finding container 3733a69e91dcea54fba6b95acbf7430687c3103cefadc5b8f0ce56a5423b3b15: Status 404 returned error can't find the container with id 3733a69e91dcea54fba6b95acbf7430687c3103cefadc5b8f0ce56a5423b3b15
Nov 02 21:29:08.967754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.967718    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:08.967849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.967804    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:08.967849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:08.967845    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:09.049643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.048000    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-xks-ca\") pod \"0db59701-54b0-4f0c-b534-eeedcf9dc520\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") "
Nov 02 21:29:09.049643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.048180    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-openshift-ca\") pod \"0db59701-54b0-4f0c-b534-eeedcf9dc520\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") "
Nov 02 21:29:09.049643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.048263    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db59701-54b0-4f0c-b534-eeedcf9dc520-kube-api-access-lxr7c\" (UniqueName: \"kubernetes.io/projected/0db59701-54b0-4f0c-b534-eeedcf9dc520-kube-api-access-lxr7c\") pod \"0db59701-54b0-4f0c-b534-eeedcf9dc520\" (UID: \"0db59701-54b0-4f0c-b534-eeedcf9dc520\") "
Nov 02 21:29:09.050817 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.050336    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-openshift-ca" pod "0db59701-54b0-4f0c-b534-eeedcf9dc520" (UID: "0db59701-54b0-4f0c-b534-eeedcf9dc520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:29:09.096649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.075599    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-xks-ca" pod "0db59701-54b0-4f0c-b534-eeedcf9dc520" (UID: "0db59701-54b0-4f0c-b534-eeedcf9dc520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:29:09.096649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.076010    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db59701-54b0-4f0c-b534-eeedcf9dc520-kube-api-access-lxr7c" pod "0db59701-54b0-4f0c-b534-eeedcf9dc520" (UID: "0db59701-54b0-4f0c-b534-eeedcf9dc520"). InnerVolumeSpecName "kube-api-access-lxr7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:29:09.149384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.149351    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:09.149384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.149381    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db59701-54b0-4f0c-b534-eeedcf9dc520-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:09.149384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.149392    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxr7c\" (UniqueName: \"kubernetes.io/projected/0db59701-54b0-4f0c-b534-eeedcf9dc520-kube-api-access-lxr7c\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:09.814135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.803859    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" event={"ID":"47468c66-6cd7-4d1e-a705-0689ed95559a","Type":"ContainerStarted","Data":"5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33"}
Nov 02 21:29:09.814135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.803959    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" event={"ID":"47468c66-6cd7-4d1e-a705-0689ed95559a","Type":"ContainerStarted","Data":"3733a69e91dcea54fba6b95acbf7430687c3103cefadc5b8f0ce56a5423b3b15"}
Nov 02 21:29:09.825519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.825476    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r" event={"ID":"0db59701-54b0-4f0c-b534-eeedcf9dc520","Type":"ContainerDied","Data":"8996ca7e0eba7a350d8b3aa1ef7d83052127deb9b983a46d68678c8ff73fffcc"}
Nov 02 21:29:09.825794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.825775    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8996ca7e0eba7a350d8b3aa1ef7d83052127deb9b983a46d68678c8ff73fffcc"
Nov 02 21:29:09.825976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.825963    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"
Nov 02 21:29:09.978636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:09.978577    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" podStartSLOduration=1.978556657 podStartE2EDuration="1.978556657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:09.964062501 +0000 UTC m=+1237.227598828" watchObservedRunningTime="2026-11-02 21:29:09.978556657 +0000 UTC m=+1237.242092975"
Nov 02 21:29:10.520134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.513493    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:10.526283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.526248    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 02 21:29:10.526456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.526308    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 02 21:29:10.782706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.782632    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"]
Nov 02 21:29:10.783172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.783157    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0db59701-54b0-4f0c-b534-eeedcf9dc520" containerName="revoke-keys"
Nov 02 21:29:10.783210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.783174    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db59701-54b0-4f0c-b534-eeedcf9dc520" containerName="revoke-keys"
Nov 02 21:29:10.783264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.783257    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db59701-54b0-4f0c-b534-eeedcf9dc520" containerName="revoke-keys"
Nov 02 21:29:10.785899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.785883    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:10.838353 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.838324    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:29:10.838543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.838437    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:29:10.838659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.838488    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:29:10.838714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.838587    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:29:10.839188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.839170    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"]
Nov 02 21:29:10.869264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.869234    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2x5\" (UniqueName: \"kubernetes.io/projected/848c527e-1850-4526-a9bf-034be4084fa9-kube-api-access-4h2x5\") pod \"maas-api-revoke-sub-75845a1a-pddtj\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:10.869388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.869294    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75845a1a-pddtj\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:10.869538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.869515    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75845a1a-pddtj\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:10.970417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.970380    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75845a1a-pddtj\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:10.970608 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.970493    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2x5\" (UniqueName: \"kubernetes.io/projected/848c527e-1850-4526-a9bf-034be4084fa9-kube-api-access-4h2x5\") pod \"maas-api-revoke-sub-75845a1a-pddtj\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:10.970608 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.970540    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75845a1a-pddtj\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:10.971085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.971062    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75845a1a-pddtj\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:10.972862 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:10.972835    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75845a1a-pddtj\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:11.028822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:11.028789    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2x5\" (UniqueName: \"kubernetes.io/projected/848c527e-1850-4526-a9bf-034be4084fa9-kube-api-access-4h2x5\") pod \"maas-api-revoke-sub-75845a1a-pddtj\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:11.099191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:11.099078    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:11.490443 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:11.490414    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848c527e_1850_4526_a9bf_034be4084fa9.slice/crio-0be3703156e822215dde6df3593e2faa18307ef3dd92f35d89a053c7c3d899ce WatchSource:0}: Error finding container 0be3703156e822215dde6df3593e2faa18307ef3dd92f35d89a053c7c3d899ce: Status 404 returned error can't find the container with id 0be3703156e822215dde6df3593e2faa18307ef3dd92f35d89a053c7c3d899ce
Nov 02 21:29:11.492410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:11.492382    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"]
Nov 02 21:29:11.524209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:11.524071    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 02 21:29:11.524209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:11.524148    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 02 21:29:11.839726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:11.839689    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj" event={"ID":"848c527e-1850-4526-a9bf-034be4084fa9","Type":"ContainerStarted","Data":"0be3703156e822215dde6df3593e2faa18307ef3dd92f35d89a053c7c3d899ce"}
Nov 02 21:29:12.520134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:12.517508    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 02 21:29:12.520134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:12.517569    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 02 21:29:12.847010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:12.846928    4528 generic.go:356] "Generic (PLEG): container finished" podID="848c527e-1850-4526-a9bf-034be4084fa9" containerID="fa1c38134473c93144133cddf9e6e9f026c677bcc028ce2808808f918a65052e" exitCode=0
Nov 02 21:29:12.847173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:12.847003    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj" event={"ID":"848c527e-1850-4526-a9bf-034be4084fa9","Type":"ContainerDied","Data":"fa1c38134473c93144133cddf9e6e9f026c677bcc028ce2808808f918a65052e"}
Nov 02 21:29:13.520134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:13.513525    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 02 21:29:13.520134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:13.513590    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 02 21:29:14.047415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.047391    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:14.130789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.130697    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/848c527e-1850-4526-a9bf-034be4084fa9-kube-api-access-4h2x5\" (UniqueName: \"kubernetes.io/projected/848c527e-1850-4526-a9bf-034be4084fa9-kube-api-access-4h2x5\") pod \"848c527e-1850-4526-a9bf-034be4084fa9\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") "
Nov 02 21:29:14.130946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.130899    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-xks-ca\") pod \"848c527e-1850-4526-a9bf-034be4084fa9\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") "
Nov 02 21:29:14.131024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.131007    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-openshift-ca\") pod \"848c527e-1850-4526-a9bf-034be4084fa9\" (UID: \"848c527e-1850-4526-a9bf-034be4084fa9\") "
Nov 02 21:29:14.131886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.131806    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-openshift-ca" pod "848c527e-1850-4526-a9bf-034be4084fa9" (UID: "848c527e-1850-4526-a9bf-034be4084fa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:29:14.132098 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:29:14.132069    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:29:14.136230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.136209    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-xks-ca" pod "848c527e-1850-4526-a9bf-034be4084fa9" (UID: "848c527e-1850-4526-a9bf-034be4084fa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:29:14.136303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.136243    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848c527e-1850-4526-a9bf-034be4084fa9-kube-api-access-4h2x5" pod "848c527e-1850-4526-a9bf-034be4084fa9" (UID: "848c527e-1850-4526-a9bf-034be4084fa9"). InnerVolumeSpecName "kube-api-access-4h2x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:29:14.233779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.233748    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:14.233952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.233941    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h2x5\" (UniqueName: \"kubernetes.io/projected/848c527e-1850-4526-a9bf-034be4084fa9-kube-api-access-4h2x5\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:14.234034 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.234027    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848c527e-1850-4526-a9bf-034be4084fa9-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:14.520132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.519499    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 02 21:29:14.520132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.519578    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 02 21:29:14.861027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.860929    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"
Nov 02 21:29:14.861027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.860928    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj" event={"ID":"848c527e-1850-4526-a9bf-034be4084fa9","Type":"ContainerDied","Data":"0be3703156e822215dde6df3593e2faa18307ef3dd92f35d89a053c7c3d899ce"}
Nov 02 21:29:14.861224 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:14.861094    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0be3703156e822215dde6df3593e2faa18307ef3dd92f35d89a053c7c3d899ce"
Nov 02 21:29:15.513550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:15.513512    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 02 21:29:15.513842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:15.513811    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 02 21:29:16.140838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.138544    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"]
Nov 02 21:29:16.140838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.139374    4528 cpu_manager.go:414] "containerMap: removing container" podUID="848c527e-1850-4526-a9bf-034be4084fa9" containerName="revoke-keys"
Nov 02 21:29:16.140838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.139396    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="848c527e-1850-4526-a9bf-034be4084fa9" containerName="revoke-keys"
Nov 02 21:29:16.140838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.139557    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="848c527e-1850-4526-a9bf-034be4084fa9" containerName="revoke-keys"
Nov 02 21:29:16.143679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.143658    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.183831 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.183800    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"]
Nov 02 21:29:16.233223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.233192    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:29:16.233534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.233516    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:29:16.300134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.278296    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:29:16.300134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.278410    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:29:16.300134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.284907    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0d88ae7-tsrpr\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.300134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.285059    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5n4b\" (UniqueName: \"kubernetes.io/projected/958fc55b-d0c8-47be-9c66-146f4570103a-kube-api-access-x5n4b\") pod \"maas-api-revoke-sub-c0d88ae7-tsrpr\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.300134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.285200    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0d88ae7-tsrpr\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.433182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.414086    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5n4b\" (UniqueName: \"kubernetes.io/projected/958fc55b-d0c8-47be-9c66-146f4570103a-kube-api-access-x5n4b\") pod \"maas-api-revoke-sub-c0d88ae7-tsrpr\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.433182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.414259    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0d88ae7-tsrpr\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.433182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.414351    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0d88ae7-tsrpr\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.433182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.415236    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0d88ae7-tsrpr\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.458605 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.458546    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0d88ae7-tsrpr\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.491703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.490769    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5n4b\" (UniqueName: \"kubernetes.io/projected/958fc55b-d0c8-47be-9c66-146f4570103a-kube-api-access-x5n4b\") pod \"maas-api-revoke-sub-c0d88ae7-tsrpr\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:16.497489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:16.497463    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:17.164132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:17.156036    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"]
Nov 02 21:29:17.519506 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:17.519461    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:29:17.519705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:17.519533    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:29:17.985423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:17.985339    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr" event={"ID":"958fc55b-d0c8-47be-9c66-146f4570103a","Type":"ContainerStarted","Data":"f795729ffabc6622e9494435fd408718d41781c3a1ab8bf5c5d3236430a69350"}
Nov 02 21:29:17.985423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:17.985395    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr" event={"ID":"958fc55b-d0c8-47be-9c66-146f4570103a","Type":"ContainerStarted","Data":"c2695ace35a1c8a5e4e9e5f2bc4c210d96b7cdcdd8b343a8c0cc93cd59390ffc"}
Nov 02 21:29:18.101317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:18.101267    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr" podStartSLOduration=2.10125428 podStartE2EDuration="2.10125428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:18.098141083 +0000 UTC m=+1245.361677401" watchObservedRunningTime="2026-11-02 21:29:18.10125428 +0000 UTC m=+1245.364790597"
Nov 02 21:29:18.343391 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:18.342245    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:18.343391 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:18.343224    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:18.344173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:18.344152    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:29:18.992652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:18.992620    4528 generic.go:356] "Generic (PLEG): container finished" podID="958fc55b-d0c8-47be-9c66-146f4570103a" containerID="f795729ffabc6622e9494435fd408718d41781c3a1ab8bf5c5d3236430a69350" exitCode=0
Nov 02 21:29:18.992817 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:18.992712    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr" event={"ID":"958fc55b-d0c8-47be-9c66-146f4570103a","Type":"ContainerDied","Data":"f795729ffabc6622e9494435fd408718d41781c3a1ab8bf5c5d3236430a69350"}
Nov 02 21:29:20.142692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.142666    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:20.223777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.223750    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958fc55b-d0c8-47be-9c66-146f4570103a-kube-api-access-x5n4b\" (UniqueName: \"kubernetes.io/projected/958fc55b-d0c8-47be-9c66-146f4570103a-kube-api-access-x5n4b\") pod \"958fc55b-d0c8-47be-9c66-146f4570103a\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") "
Nov 02 21:29:20.223930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.223821    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-openshift-ca\") pod \"958fc55b-d0c8-47be-9c66-146f4570103a\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") "
Nov 02 21:29:20.223930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.223865    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-xks-ca\") pod \"958fc55b-d0c8-47be-9c66-146f4570103a\" (UID: \"958fc55b-d0c8-47be-9c66-146f4570103a\") "
Nov 02 21:29:20.224245 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.224215    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-openshift-ca" pod "958fc55b-d0c8-47be-9c66-146f4570103a" (UID: "958fc55b-d0c8-47be-9c66-146f4570103a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:29:20.225827 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.225789    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958fc55b-d0c8-47be-9c66-146f4570103a-kube-api-access-x5n4b" pod "958fc55b-d0c8-47be-9c66-146f4570103a" (UID: "958fc55b-d0c8-47be-9c66-146f4570103a"). InnerVolumeSpecName "kube-api-access-x5n4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:29:20.225956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.225880    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-xks-ca" pod "958fc55b-d0c8-47be-9c66-146f4570103a" (UID: "958fc55b-d0c8-47be-9c66-146f4570103a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:29:20.325406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.325372    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:20.325406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.325404    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5n4b\" (UniqueName: \"kubernetes.io/projected/958fc55b-d0c8-47be-9c66-146f4570103a-kube-api-access-x5n4b\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:20.325609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:20.325418    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958fc55b-d0c8-47be-9c66-146f4570103a-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:21.005636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:21.005606    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"
Nov 02 21:29:21.005803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:21.005647    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr" event={"ID":"958fc55b-d0c8-47be-9c66-146f4570103a","Type":"ContainerDied","Data":"c2695ace35a1c8a5e4e9e5f2bc4c210d96b7cdcdd8b343a8c0cc93cd59390ffc"}
Nov 02 21:29:21.005803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:21.005696    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2695ace35a1c8a5e4e9e5f2bc4c210d96b7cdcdd8b343a8c0cc93cd59390ffc"
Nov 02 21:29:21.051663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:21.051629    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"]
Nov 02 21:29:21.102041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:21.102010    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb9c929-thsvk"]
Nov 02 21:29:22.912860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:22.912827    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc86f8b1-623c-4bad-853f-39b58ee0af44" path="/var/lib/kubelet/pods/fc86f8b1-623c-4bad-853f-39b58ee0af44/volumes"
Nov 02 21:29:30.394659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:30.394624    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:29:33.255236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.255205    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"]
Nov 02 21:29:33.255701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.255686    4528 cpu_manager.go:414] "containerMap: removing container" podUID="958fc55b-d0c8-47be-9c66-146f4570103a" containerName="revoke-keys"
Nov 02 21:29:33.255701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.255701    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958fc55b-d0c8-47be-9c66-146f4570103a" containerName="revoke-keys"
Nov 02 21:29:33.255802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.255793    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="958fc55b-d0c8-47be-9c66-146f4570103a" containerName="revoke-keys"
Nov 02 21:29:33.260104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.260085    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.311602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.311573    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"]
Nov 02 21:29:33.326055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.326020    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad87a75-557sz\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.326245 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.326226    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2fd\" (UniqueName: \"kubernetes.io/projected/4b7d3cff-9468-421b-acdd-812f2f79c48b-kube-api-access-wv2fd\") pod \"maas-api-revoke-sub-8ad87a75-557sz\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.326317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.326282    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad87a75-557sz\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.358398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.358370    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:29:33.358398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.358399    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:29:33.362311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.362289    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:29:33.362311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.362295    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:29:33.427275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.427243    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad87a75-557sz\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.427452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.427376    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2fd\" (UniqueName: \"kubernetes.io/projected/4b7d3cff-9468-421b-acdd-812f2f79c48b-kube-api-access-wv2fd\") pod \"maas-api-revoke-sub-8ad87a75-557sz\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.427452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.427426    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad87a75-557sz\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.427891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.427870    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad87a75-557sz\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.429720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.429697    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad87a75-557sz\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.484775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.484747    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2fd\" (UniqueName: \"kubernetes.io/projected/4b7d3cff-9468-421b-acdd-812f2f79c48b-kube-api-access-wv2fd\") pod \"maas-api-revoke-sub-8ad87a75-557sz\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.573325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.573292    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:33.940132 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:33.940087    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7d3cff_9468_421b_acdd_812f2f79c48b.slice/crio-5302dd247227c064b2787c7eb822a23ba968a779bb54ef16693629d6a32b086b WatchSource:0}: Error finding container 5302dd247227c064b2787c7eb822a23ba968a779bb54ef16693629d6a32b086b: Status 404 returned error can't find the container with id 5302dd247227c064b2787c7eb822a23ba968a779bb54ef16693629d6a32b086b
Nov 02 21:29:33.944055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:33.944032    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"]
Nov 02 21:29:34.112587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:34.112555    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz" event={"ID":"4b7d3cff-9468-421b-acdd-812f2f79c48b","Type":"ContainerStarted","Data":"5302dd247227c064b2787c7eb822a23ba968a779bb54ef16693629d6a32b086b"}
Nov 02 21:29:35.120026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:35.119941    4528 generic.go:356] "Generic (PLEG): container finished" podID="4b7d3cff-9468-421b-acdd-812f2f79c48b" containerID="9ce150e9d4f9d2219537f17e889bff272f11fdbf7a79749bd7322fd8ff5d5714" exitCode=0
Nov 02 21:29:35.120386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:35.120028    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz" event={"ID":"4b7d3cff-9468-421b-acdd-812f2f79c48b","Type":"ContainerDied","Data":"9ce150e9d4f9d2219537f17e889bff272f11fdbf7a79749bd7322fd8ff5d5714"}
Nov 02 21:29:36.275771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.275746    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:36.359905 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.359874    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7d3cff-9468-421b-acdd-812f2f79c48b-kube-api-access-wv2fd\" (UniqueName: \"kubernetes.io/projected/4b7d3cff-9468-421b-acdd-812f2f79c48b-kube-api-access-wv2fd\") pod \"4b7d3cff-9468-421b-acdd-812f2f79c48b\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") "
Nov 02 21:29:36.360052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.359971    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-xks-ca\") pod \"4b7d3cff-9468-421b-acdd-812f2f79c48b\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") "
Nov 02 21:29:36.360052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.360038    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-openshift-ca\") pod \"4b7d3cff-9468-421b-acdd-812f2f79c48b\" (UID: \"4b7d3cff-9468-421b-acdd-812f2f79c48b\") "
Nov 02 21:29:36.360344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.360319    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-openshift-ca" pod "4b7d3cff-9468-421b-acdd-812f2f79c48b" (UID: "4b7d3cff-9468-421b-acdd-812f2f79c48b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:29:36.360500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.360485    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:36.361906 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.361873    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7d3cff-9468-421b-acdd-812f2f79c48b-kube-api-access-wv2fd" pod "4b7d3cff-9468-421b-acdd-812f2f79c48b" (UID: "4b7d3cff-9468-421b-acdd-812f2f79c48b"). InnerVolumeSpecName "kube-api-access-wv2fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:29:36.362032 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.361945    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-xks-ca" pod "4b7d3cff-9468-421b-acdd-812f2f79c48b" (UID: "4b7d3cff-9468-421b-acdd-812f2f79c48b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:29:36.461665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.461584    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2fd\" (UniqueName: \"kubernetes.io/projected/4b7d3cff-9468-421b-acdd-812f2f79c48b-kube-api-access-wv2fd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:36.461665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:36.461625    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b7d3cff-9468-421b-acdd-812f2f79c48b-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:37.132658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:37.132629    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"
Nov 02 21:29:37.132837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:37.132674    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz" event={"ID":"4b7d3cff-9468-421b-acdd-812f2f79c48b","Type":"ContainerDied","Data":"5302dd247227c064b2787c7eb822a23ba968a779bb54ef16693629d6a32b086b"}
Nov 02 21:29:37.132837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:37.132725    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5302dd247227c064b2787c7eb822a23ba968a779bb54ef16693629d6a32b086b"
Nov 02 21:29:38.048487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.048448    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"]
Nov 02 21:29:38.106527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.100680    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e1b58c-zcj8w"]
Nov 02 21:29:38.804329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.804296    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"]
Nov 02 21:29:38.804811 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.804798    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4b7d3cff-9468-421b-acdd-812f2f79c48b" containerName="revoke-keys"
Nov 02 21:29:38.804866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.804812    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7d3cff-9468-421b-acdd-812f2f79c48b" containerName="revoke-keys"
Nov 02 21:29:38.804925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.804917    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7d3cff-9468-421b-acdd-812f2f79c48b" containerName="revoke-keys"
Nov 02 21:29:38.808314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.808298    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:38.861285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.861245    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"]
Nov 02 21:29:38.885861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.885800    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-435eae3f-9bxgt\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:38.886048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.885984    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-435eae3f-9bxgt\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:38.886166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.886144    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2c9\" (UniqueName: \"kubernetes.io/projected/af239596-ac2a-4136-97d2-db32ad4808ec-kube-api-access-tj2c9\") pod \"maas-api-revoke-sub-435eae3f-9bxgt\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:38.907941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.907917    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:29:38.907941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.907929    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:29:38.907941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.907927    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:29:38.910537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.910518    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:29:38.913220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.913199    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ceaee2-7bc0-4fcc-b83c-75595a5548be" path="/var/lib/kubelet/pods/c8ceaee2-7bc0-4fcc-b83c-75595a5548be/volumes"
Nov 02 21:29:38.986961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.986925    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-435eae3f-9bxgt\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:38.987155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.987025    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-435eae3f-9bxgt\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:38.987155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.987139    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2c9\" (UniqueName: \"kubernetes.io/projected/af239596-ac2a-4136-97d2-db32ad4808ec-kube-api-access-tj2c9\") pod \"maas-api-revoke-sub-435eae3f-9bxgt\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:38.987619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.987596    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-435eae3f-9bxgt\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:38.989331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:38.989311    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-435eae3f-9bxgt\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:39.044300 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.044266    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2c9\" (UniqueName: \"kubernetes.io/projected/af239596-ac2a-4136-97d2-db32ad4808ec-kube-api-access-tj2c9\") pod \"maas-api-revoke-sub-435eae3f-9bxgt\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:39.047291 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.047267    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"]
Nov 02 21:29:39.050307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.050280    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a5c342-ws9ml"]
Nov 02 21:29:39.121532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.121462    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:39.494853 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:39.494820    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf239596_ac2a_4136_97d2_db32ad4808ec.slice/crio-c3ab5cde142939d508771e5b9b5ff1c85316ce8e9198707411b87d0cc2d608eb WatchSource:0}: Error finding container c3ab5cde142939d508771e5b9b5ff1c85316ce8e9198707411b87d0cc2d608eb: Status 404 returned error can't find the container with id c3ab5cde142939d508771e5b9b5ff1c85316ce8e9198707411b87d0cc2d608eb
Nov 02 21:29:39.498158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.498131    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"]
Nov 02 21:29:39.601768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.601733    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"]
Nov 02 21:29:39.607042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.607026    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.659508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.659481    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0c9014-openshift-default-dockercfg-7ll9g\""
Nov 02 21:29:39.696624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.696591    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.696767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.696639    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6chql\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-kube-api-access-6chql\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.696767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.696684    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.696767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.696726    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.696882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.696769    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.696882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.696821    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.697129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.697086    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d258233-0e4a-48ae-a387-94c13aa4c401-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.697204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.697146    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.697243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.697222    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.712804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.712776    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"]
Nov 02 21:29:39.798700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.798671    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.798700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.798704    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6chql\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-kube-api-access-6chql\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.798902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.798734    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.798902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.798760    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.798902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.798780    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.798902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.798808    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.799069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.798924    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d258233-0e4a-48ae-a387-94c13aa4c401-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.799069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.798972    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.799069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.799029    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.799289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.799262    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.799451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.799430    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.799540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.799478    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.799540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.799518    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.799734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.799712    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d258233-0e4a-48ae-a387-94c13aa4c401-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.801142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.801121    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.801366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.801351    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.855959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.855931    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.856290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.856267    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6chql\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-kube-api-access-6chql\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:39.918988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:39.918960    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:40.153779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:40.153690    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt" event={"ID":"af239596-ac2a-4136-97d2-db32ad4808ec","Type":"ContainerStarted","Data":"9c0669bf0835586c12ca73ffba353c35d13da0a0ef2eca8e77ad237254b89d67"}
Nov 02 21:29:40.153779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:40.153755    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt" event={"ID":"af239596-ac2a-4136-97d2-db32ad4808ec","Type":"ContainerStarted","Data":"c3ab5cde142939d508771e5b9b5ff1c85316ce8e9198707411b87d0cc2d608eb"}
Nov 02 21:29:40.269642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:40.269586    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt" podStartSLOduration=2.269568046 podStartE2EDuration="2.269568046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:40.266865513 +0000 UTC m=+1267.530401828" watchObservedRunningTime="2026-11-02 21:29:40.269568046 +0000 UTC m=+1267.533104363"
Nov 02 21:29:40.424363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:40.424333    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"]
Nov 02 21:29:40.466009 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:40.465973    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d258233_0e4a_48ae_a387_94c13aa4c401.slice/crio-0461f96b026082c7e213f2096df87d50376a8a69d3d96637901a145815cf65ea WatchSource:0}: Error finding container 0461f96b026082c7e213f2096df87d50376a8a69d3d96637901a145815cf65ea: Status 404 returned error can't find the container with id 0461f96b026082c7e213f2096df87d50376a8a69d3d96637901a145815cf65ea
Nov 02 21:29:40.468514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:40.468486    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:40.468588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:40.468551    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:40.468588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:40.468581    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:40.913454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:40.913420    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff28f90a-5574-4eb2-bfff-bc905d6be61f" path="/var/lib/kubelet/pods/ff28f90a-5574-4eb2-bfff-bc905d6be61f/volumes"
Nov 02 21:29:41.168393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.168297    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" event={"ID":"9d258233-0e4a-48ae-a387-94c13aa4c401","Type":"ContainerStarted","Data":"cb487527b8df7540a004c089458bb2195c5bca3a62f9d8ddddc06a19f74b9961"}
Nov 02 21:29:41.168393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.168384    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" event={"ID":"9d258233-0e4a-48ae-a387-94c13aa4c401","Type":"ContainerStarted","Data":"0461f96b026082c7e213f2096df87d50376a8a69d3d96637901a145815cf65ea"}
Nov 02 21:29:41.169941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.169917    4528 generic.go:356] "Generic (PLEG): container finished" podID="af239596-ac2a-4136-97d2-db32ad4808ec" containerID="9c0669bf0835586c12ca73ffba353c35d13da0a0ef2eca8e77ad237254b89d67" exitCode=0
Nov 02 21:29:41.170049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.169979    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt" event={"ID":"af239596-ac2a-4136-97d2-db32ad4808ec","Type":"ContainerDied","Data":"9c0669bf0835586c12ca73ffba353c35d13da0a0ef2eca8e77ad237254b89d67"}
Nov 02 21:29:41.296889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.296834    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" podStartSLOduration=2.296820591 podStartE2EDuration="2.296820591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:41.294419945 +0000 UTC m=+1268.557956265" watchObservedRunningTime="2026-11-02 21:29:41.296820591 +0000 UTC m=+1268.560356972"
Nov 02 21:29:41.637931 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.637889    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"]
Nov 02 21:29:41.642119 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.642082    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.738791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.738175    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"]
Nov 02 21:29:41.834763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.834728    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1698d6b6-be71-4474-8747-2fc8a484770a-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.834763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.834767    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.835001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.834790    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.835001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.834896    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.835001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.834925    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnmb\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-kube-api-access-7fnmb\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.835151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.835026    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.835151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.835050    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1698d6b6-be71-4474-8747-2fc8a484770a-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.835151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.835077    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.835151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.835145    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.925182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.924291    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:41.927731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.927701    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 21:29:41.927850 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.927760    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 21:29:41.935761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.935736    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.935821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.935777    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnmb\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-kube-api-access-7fnmb\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.935903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.935882    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.935936    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1698d6b6-be71-4474-8747-2fc8a484770a-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.935987    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.936042    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.936090    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1698d6b6-be71-4474-8747-2fc8a484770a-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.936148    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.936175    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.936284    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.936425    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.936659    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1698d6b6-be71-4474-8747-2fc8a484770a-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.936800    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.936928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.936824    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.938598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.938572    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.938708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.938691    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1698d6b6-be71-4474-8747-2fc8a484770a-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.992422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.992388    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:41.992586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:41.992451    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnmb\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-kube-api-access-7fnmb\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:42.255146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.254977    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:42.277074 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.277032    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"]
Nov 02 21:29:42.282412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.282390    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.338813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.338779    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"]
Nov 02 21:29:42.341288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.339953    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3408d1-58thw\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.341288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.340066    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3408d1-58thw\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.341288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.340201    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjd6k\" (UniqueName: \"kubernetes.io/projected/a6201e1a-34ee-4c94-b43a-9fc083e98810-kube-api-access-xjd6k\") pod \"maas-api-revoke-sub-9f3408d1-58thw\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.360410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.360384    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:42.440957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.440872    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af239596-ac2a-4136-97d2-db32ad4808ec-kube-api-access-tj2c9\" (UniqueName: \"kubernetes.io/projected/af239596-ac2a-4136-97d2-db32ad4808ec-kube-api-access-tj2c9\") pod \"af239596-ac2a-4136-97d2-db32ad4808ec\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") "
Nov 02 21:29:42.440957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.440923    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-xks-ca\") pod \"af239596-ac2a-4136-97d2-db32ad4808ec\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") "
Nov 02 21:29:42.441179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.440966    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-openshift-ca\") pod \"af239596-ac2a-4136-97d2-db32ad4808ec\" (UID: \"af239596-ac2a-4136-97d2-db32ad4808ec\") "
Nov 02 21:29:42.441288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.441267    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3408d1-58thw\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.441387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.441369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjd6k\" (UniqueName: \"kubernetes.io/projected/a6201e1a-34ee-4c94-b43a-9fc083e98810-kube-api-access-xjd6k\") pod \"maas-api-revoke-sub-9f3408d1-58thw\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.441490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.441425    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-openshift-ca" pod "af239596-ac2a-4136-97d2-db32ad4808ec" (UID: "af239596-ac2a-4136-97d2-db32ad4808ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:29:42.441490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.441454    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3408d1-58thw\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.441580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.441535    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:42.442276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.442251    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3408d1-58thw\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.443706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.443685    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-xks-ca" pod "af239596-ac2a-4136-97d2-db32ad4808ec" (UID: "af239596-ac2a-4136-97d2-db32ad4808ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:29:42.444181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.444148    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af239596-ac2a-4136-97d2-db32ad4808ec-kube-api-access-tj2c9" pod "af239596-ac2a-4136-97d2-db32ad4808ec" (UID: "af239596-ac2a-4136-97d2-db32ad4808ec"). InnerVolumeSpecName "kube-api-access-tj2c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:29:42.444547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.444530    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3408d1-58thw\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.498503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.498483    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjd6k\" (UniqueName: \"kubernetes.io/projected/a6201e1a-34ee-4c94-b43a-9fc083e98810-kube-api-access-xjd6k\") pod \"maas-api-revoke-sub-9f3408d1-58thw\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.543003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.542941    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj2c9\" (UniqueName: \"kubernetes.io/projected/af239596-ac2a-4136-97d2-db32ad4808ec-kube-api-access-tj2c9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:42.543003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.542978    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af239596-ac2a-4136-97d2-db32ad4808ec-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:42.646849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.646821    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:42.741925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.741892    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"]
Nov 02 21:29:42.743822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.743793    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:42.744011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.743990    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:42.744185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.744169    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:42.934586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.934415    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 21:29:42.934586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:42.934470    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 21:29:43.031704 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.031666    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"]
Nov 02 21:29:43.184890 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.184862    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw" event={"ID":"a6201e1a-34ee-4c94-b43a-9fc083e98810","Type":"ContainerStarted","Data":"6717c2f1f52518691f07efc21083c9ea837505b994c95b88f0618624d08ad85c"}
Nov 02 21:29:43.186709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.186681    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" event={"ID":"1698d6b6-be71-4474-8747-2fc8a484770a","Type":"ContainerStarted","Data":"6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637"}
Nov 02 21:29:43.186825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.186727    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" event={"ID":"1698d6b6-be71-4474-8747-2fc8a484770a","Type":"ContainerStarted","Data":"a61e4aeceffc858df3036b6682dca9c84788da37fe6197e09e4858e8c647fece"}
Nov 02 21:29:43.189428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.189403    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"
Nov 02 21:29:43.189654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.189628    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt" event={"ID":"af239596-ac2a-4136-97d2-db32ad4808ec","Type":"ContainerDied","Data":"c3ab5cde142939d508771e5b9b5ff1c85316ce8e9198707411b87d0cc2d608eb"}
Nov 02 21:29:43.189739 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.189694    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3ab5cde142939d508771e5b9b5ff1c85316ce8e9198707411b87d0cc2d608eb"
Nov 02 21:29:43.256417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.256390    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:43.257366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.257345    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 21:29:43.257438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.257389    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 21:29:43.331339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.331034    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" podStartSLOduration=2.331014027 podStartE2EDuration="2.331014027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:43.328832123 +0000 UTC m=+1270.592368441" watchObservedRunningTime="2026-11-02 21:29:43.331014027 +0000 UTC m=+1270.594550345"
Nov 02 21:29:43.926135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.925517    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 21:29:43.926135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:43.925587    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 21:29:44.055282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.055247    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"]
Nov 02 21:29:44.056286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.056264    4528 cpu_manager.go:414] "containerMap: removing container" podUID="af239596-ac2a-4136-97d2-db32ad4808ec" containerName="revoke-keys"
Nov 02 21:29:44.056430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.056419    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af239596-ac2a-4136-97d2-db32ad4808ec" containerName="revoke-keys"
Nov 02 21:29:44.056642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.056623    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="af239596-ac2a-4136-97d2-db32ad4808ec" containerName="revoke-keys"
Nov 02 21:29:44.060181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.060164    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.061222 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.061205    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"]
Nov 02 21:29:44.066167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.065776    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604c3c35-q6t8c"]
Nov 02 21:29:44.067161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.067135    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16a2f95-mmdmf\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.067251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.067172    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdsj\" (UniqueName: \"kubernetes.io/projected/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-kube-api-access-dmdsj\") pod \"maas-api-revoke-sub-e16a2f95-mmdmf\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.067331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.067269    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16a2f95-mmdmf\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.095700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.095634    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"]
Nov 02 21:29:44.166069 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:29:44.166032    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:29:44.169474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.169445    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16a2f95-mmdmf\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.169629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.169496    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdsj\" (UniqueName: \"kubernetes.io/projected/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-kube-api-access-dmdsj\") pod \"maas-api-revoke-sub-e16a2f95-mmdmf\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.169629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.169542    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16a2f95-mmdmf\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.170211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.170178    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16a2f95-mmdmf\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.175718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.175692    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16a2f95-mmdmf\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.198320 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.198256    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw" event={"ID":"a6201e1a-34ee-4c94-b43a-9fc083e98810","Type":"ContainerStarted","Data":"ac4d62a800edbb22fc4baee46ea19d1b3c59f8d2d8d433c1a5dd9a20a3747841"}
Nov 02 21:29:44.254072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.254037    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdsj\" (UniqueName: \"kubernetes.io/projected/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-kube-api-access-dmdsj\") pod \"maas-api-revoke-sub-e16a2f95-mmdmf\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.255869 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.255831    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 21:29:44.255919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.255903    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 21:29:44.354285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.353434    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw" podStartSLOduration=2.3534104940000002 podStartE2EDuration="2.353410494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:44.32829288 +0000 UTC m=+1271.591829199" watchObservedRunningTime="2026-11-02 21:29:44.353410494 +0000 UTC m=+1271.616946832"
Nov 02 21:29:44.374284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.374257    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:44.791903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.791834    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"]
Nov 02 21:29:44.942139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.941208    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 21:29:44.942139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.941264    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 21:29:44.956469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:44.956423    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c7b2bc-0c36-46db-982b-b6692dd039db" path="/var/lib/kubelet/pods/89c7b2bc-0c36-46db-982b-b6692dd039db/volumes"
Nov 02 21:29:45.230135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:45.226220    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf" event={"ID":"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4","Type":"ContainerStarted","Data":"dc62999f670ee3e35b6baea9ad789bef33447589a794a95189d21a12fca45dcb"}
Nov 02 21:29:45.238183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:45.237640    4528 generic.go:356] "Generic (PLEG): container finished" podID="a6201e1a-34ee-4c94-b43a-9fc083e98810" containerID="ac4d62a800edbb22fc4baee46ea19d1b3c59f8d2d8d433c1a5dd9a20a3747841" exitCode=0
Nov 02 21:29:45.238183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:45.237965    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw" event={"ID":"a6201e1a-34ee-4c94-b43a-9fc083e98810","Type":"ContainerDied","Data":"ac4d62a800edbb22fc4baee46ea19d1b3c59f8d2d8d433c1a5dd9a20a3747841"}
Nov 02 21:29:45.266140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:45.262389    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 21:29:45.266140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:45.262447    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 21:29:45.922008 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:45.921491    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 21:29:45.922008 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:45.921551    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 21:29:46.255467 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.255429    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 21:29:46.255670 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.255486    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 21:29:46.257272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.257243    4528 generic.go:356] "Generic (PLEG): container finished" podID="c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4" containerID="af83187215c733ad711a9320f640dbd8b92ae149cff8fb41bd29fc174739582e" exitCode=0
Nov 02 21:29:46.257799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.257779    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf" event={"ID":"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4","Type":"ContainerDied","Data":"af83187215c733ad711a9320f640dbd8b92ae149cff8fb41bd29fc174739582e"}
Nov 02 21:29:46.486432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.486405    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:46.637993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.637806    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-xks-ca\") pod \"a6201e1a-34ee-4c94-b43a-9fc083e98810\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") "
Nov 02 21:29:46.637993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.637869    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6201e1a-34ee-4c94-b43a-9fc083e98810-kube-api-access-xjd6k\" (UniqueName: \"kubernetes.io/projected/a6201e1a-34ee-4c94-b43a-9fc083e98810-kube-api-access-xjd6k\") pod \"a6201e1a-34ee-4c94-b43a-9fc083e98810\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") "
Nov 02 21:29:46.637993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.637905    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-openshift-ca\") pod \"a6201e1a-34ee-4c94-b43a-9fc083e98810\" (UID: \"a6201e1a-34ee-4c94-b43a-9fc083e98810\") "
Nov 02 21:29:46.638609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.638583    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-openshift-ca" pod "a6201e1a-34ee-4c94-b43a-9fc083e98810" (UID: "a6201e1a-34ee-4c94-b43a-9fc083e98810"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:29:46.640685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.640650    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6201e1a-34ee-4c94-b43a-9fc083e98810-kube-api-access-xjd6k" pod "a6201e1a-34ee-4c94-b43a-9fc083e98810" (UID: "a6201e1a-34ee-4c94-b43a-9fc083e98810"). InnerVolumeSpecName "kube-api-access-xjd6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:29:46.640788 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.640738    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-xks-ca" pod "a6201e1a-34ee-4c94-b43a-9fc083e98810" (UID: "a6201e1a-34ee-4c94-b43a-9fc083e98810"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:29:46.739623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.739527    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:46.739623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.739563    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjd6k\" (UniqueName: \"kubernetes.io/projected/a6201e1a-34ee-4c94-b43a-9fc083e98810-kube-api-access-xjd6k\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:46.739623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.739578    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6201e1a-34ee-4c94-b43a-9fc083e98810-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:46.928459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.928422    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 21:29:46.929042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:46.929012    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 21:29:47.255691 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.255652    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 21:29:47.255896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.255708    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 21:29:47.264741 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.264713    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"
Nov 02 21:29:47.264845 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.264736    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw" event={"ID":"a6201e1a-34ee-4c94-b43a-9fc083e98810","Type":"ContainerDied","Data":"6717c2f1f52518691f07efc21083c9ea837505b994c95b88f0618624d08ad85c"}
Nov 02 21:29:47.264845 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.264788    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6717c2f1f52518691f07efc21083c9ea837505b994c95b88f0618624d08ad85c"
Nov 02 21:29:47.468797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.468771    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:47.551909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.551881    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-openshift-ca\") pod \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") "
Nov 02 21:29:47.552141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.551977    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-xks-ca\") pod \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") "
Nov 02 21:29:47.552141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.552061    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-kube-api-access-dmdsj\" (UniqueName: \"kubernetes.io/projected/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-kube-api-access-dmdsj\") pod \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\" (UID: \"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4\") "
Nov 02 21:29:47.552337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.552313    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-openshift-ca" pod "c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4" (UID: "c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:29:47.552672 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.552648    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:47.554146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.554101    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-xks-ca" pod "c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4" (UID: "c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:29:47.554313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.554292    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-kube-api-access-dmdsj" pod "c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4" (UID: "c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4"). InnerVolumeSpecName "kube-api-access-dmdsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:29:47.654702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.654654    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:47.654901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.654732    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdsj\" (UniqueName: \"kubernetes.io/projected/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4-kube-api-access-dmdsj\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:47.965236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.965154    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:47.965752 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.965643    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:47.966440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:47.966422    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:29:48.259964 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:48.257482    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 21:29:48.259964 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:48.257536    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 21:29:48.272611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:48.272580    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"
Nov 02 21:29:48.272791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:48.272658    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf" event={"ID":"c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4","Type":"ContainerDied","Data":"dc62999f670ee3e35b6baea9ad789bef33447589a794a95189d21a12fca45dcb"}
Nov 02 21:29:48.272791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:48.272711    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc62999f670ee3e35b6baea9ad789bef33447589a794a95189d21a12fca45dcb"
Nov 02 21:29:49.256156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.256124    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 21:29:49.256650 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.256177    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 21:29:49.559539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.559504    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"]
Nov 02 21:29:49.559999 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.559986    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a6201e1a-34ee-4c94-b43a-9fc083e98810" containerName="revoke-keys"
Nov 02 21:29:49.560038 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.560000    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6201e1a-34ee-4c94-b43a-9fc083e98810" containerName="revoke-keys"
Nov 02 21:29:49.560038 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.560017    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4" containerName="revoke-keys"
Nov 02 21:29:49.560038 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.560022    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4" containerName="revoke-keys"
Nov 02 21:29:49.560161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.560105    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6201e1a-34ee-4c94-b43a-9fc083e98810" containerName="revoke-keys"
Nov 02 21:29:49.560161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.560130    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4" containerName="revoke-keys"
Nov 02 21:29:49.563730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.563714    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.615758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.615729    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"]
Nov 02 21:29:49.662352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.662312    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:29:49.662352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.662335    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:29:49.662352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.662316    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:29:49.666183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.666158    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:29:49.679622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.679593    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v9z7\" (UniqueName: \"kubernetes.io/projected/1ec79f71-7b45-4942-a5b2-a6453ce198cf-kube-api-access-2v9z7\") pod \"maas-api-revoke-sub-d46d4b76-6pfhm\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.679796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.679773    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d46d4b76-6pfhm\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.679919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.679898    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d46d4b76-6pfhm\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.780965 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.780923    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d46d4b76-6pfhm\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.781150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.781018    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d46d4b76-6pfhm\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.781150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.781070    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v9z7\" (UniqueName: \"kubernetes.io/projected/1ec79f71-7b45-4942-a5b2-a6453ce198cf-kube-api-access-2v9z7\") pod \"maas-api-revoke-sub-d46d4b76-6pfhm\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.781760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.781736    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d46d4b76-6pfhm\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.783527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.783505    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d46d4b76-6pfhm\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.838180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.838093    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v9z7\" (UniqueName: \"kubernetes.io/projected/1ec79f71-7b45-4942-a5b2-a6453ce198cf-kube-api-access-2v9z7\") pod \"maas-api-revoke-sub-d46d4b76-6pfhm\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:49.878849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:49.878815    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:50.259985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:50.259917    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:50.260359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:50.260329    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:50.261136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:50.261092    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:29:50.277807 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:50.277757    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec79f71_7b45_4942_a5b2_a6453ce198cf.slice/crio-957f8e3cefc5b82b8b953114a4b472823c9b912b0e873aefcd0dda0bee85b8bd WatchSource:0}: Error finding container 957f8e3cefc5b82b8b953114a4b472823c9b912b0e873aefcd0dda0bee85b8bd: Status 404 returned error can't find the container with id 957f8e3cefc5b82b8b953114a4b472823c9b912b0e873aefcd0dda0bee85b8bd
Nov 02 21:29:50.281309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:50.281014    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"]
Nov 02 21:29:50.292899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:50.292865    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm" event={"ID":"1ec79f71-7b45-4942-a5b2-a6453ce198cf","Type":"ContainerStarted","Data":"957f8e3cefc5b82b8b953114a4b472823c9b912b0e873aefcd0dda0bee85b8bd"}
Nov 02 21:29:51.300661 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:51.300631    4528 generic.go:356] "Generic (PLEG): container finished" podID="1ec79f71-7b45-4942-a5b2-a6453ce198cf" containerID="009b5fd45047372c2fb0b3c91774e0d853a6f3c6f9a57e69f112f40d3a37c0b6" exitCode=0
Nov 02 21:29:51.301091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:51.300714    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm" event={"ID":"1ec79f71-7b45-4942-a5b2-a6453ce198cf","Type":"ContainerDied","Data":"009b5fd45047372c2fb0b3c91774e0d853a6f3c6f9a57e69f112f40d3a37c0b6"}
Nov 02 21:29:52.459064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.459038    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:52.633884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.633809    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec79f71-7b45-4942-a5b2-a6453ce198cf-kube-api-access-2v9z7\" (UniqueName: \"kubernetes.io/projected/1ec79f71-7b45-4942-a5b2-a6453ce198cf-kube-api-access-2v9z7\") pod \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") "
Nov 02 21:29:52.633884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.633864    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-openshift-ca\") pod \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") "
Nov 02 21:29:52.634057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.633981    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-xks-ca\") pod \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\" (UID: \"1ec79f71-7b45-4942-a5b2-a6453ce198cf\") "
Nov 02 21:29:52.634285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.634261    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-openshift-ca" pod "1ec79f71-7b45-4942-a5b2-a6453ce198cf" (UID: "1ec79f71-7b45-4942-a5b2-a6453ce198cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:29:52.634568 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.634546    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:52.635955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.635931    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-xks-ca" pod "1ec79f71-7b45-4942-a5b2-a6453ce198cf" (UID: "1ec79f71-7b45-4942-a5b2-a6453ce198cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:29:52.636087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.636068    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec79f71-7b45-4942-a5b2-a6453ce198cf-kube-api-access-2v9z7" pod "1ec79f71-7b45-4942-a5b2-a6453ce198cf" (UID: "1ec79f71-7b45-4942-a5b2-a6453ce198cf"). InnerVolumeSpecName "kube-api-access-2v9z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:29:52.736247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.736217    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v9z7\" (UniqueName: \"kubernetes.io/projected/1ec79f71-7b45-4942-a5b2-a6453ce198cf-kube-api-access-2v9z7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:52.736247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:52.736242    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec79f71-7b45-4942-a5b2-a6453ce198cf-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:29:53.314786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:53.314749    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm" event={"ID":"1ec79f71-7b45-4942-a5b2-a6453ce198cf","Type":"ContainerDied","Data":"957f8e3cefc5b82b8b953114a4b472823c9b912b0e873aefcd0dda0bee85b8bd"}
Nov 02 21:29:53.314786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:53.314776    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"
Nov 02 21:29:53.314984 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:53.314810    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="957f8e3cefc5b82b8b953114a4b472823c9b912b0e873aefcd0dda0bee85b8bd"
Nov 02 21:29:54.062132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:54.061474    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"]
Nov 02 21:29:54.113690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:54.113645    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c03c89-wh678"]
Nov 02 21:29:54.930989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:54.930948    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feacee19-1354-434e-a316-f4f1bd8e41f6" path="/var/lib/kubelet/pods/feacee19-1354-434e-a316-f4f1bd8e41f6/volumes"
Nov 02 21:29:55.067134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:55.065209    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"]
Nov 02 21:29:55.108177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:55.108084    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb4183-q8b2n"]
Nov 02 21:29:55.892286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:55.892233    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:29:55.936058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:55.936008    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:29:56.039358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:56.022309    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:29:56.155192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:56.153201    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:29:56.757250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:56.757202    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:29:56.765240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:56.765198    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:29:56.899309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:56.899258    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:29:56.900512 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:56.900477    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:29:56.936599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:56.936563    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a1c4c9-a5ca-4992-b21b-41c774df7df4" path="/var/lib/kubelet/pods/f6a1c4c9-a5ca-4992-b21b-41c774df7df4/volumes"
Nov 02 21:29:56.947383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:56.947334    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 02 21:29:56.948162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:56.948092    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:29:57.023055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.023016    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:29:57.023935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.023912    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:29:57.199164 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.185050    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 02 21:29:57.209133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.204081    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:29:57.213135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.209409    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:29:57.252198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.252142    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 21:29:57.305875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.305797    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:29:57.368142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.368037    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:29:57.939068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.939036    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"]
Nov 02 21:29:57.939652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.939636    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1ec79f71-7b45-4942-a5b2-a6453ce198cf" containerName="revoke-keys"
Nov 02 21:29:57.939696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.939654    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec79f71-7b45-4942-a5b2-a6453ce198cf" containerName="revoke-keys"
Nov 02 21:29:57.939750 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.939741    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec79f71-7b45-4942-a5b2-a6453ce198cf" containerName="revoke-keys"
Nov 02 21:29:57.944448 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.944424    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:57.951663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.951639    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc785ce6-lnv7v\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:57.951859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.951839    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kctc\" (UniqueName: \"kubernetes.io/projected/0b0c3f31-1d14-4535-bfc5-4af2e0233682-kube-api-access-8kctc\") pod \"maas-api-revoke-sub-bc785ce6-lnv7v\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:57.951943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.951926    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc785ce6-lnv7v\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:57.995408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.995380    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"]
Nov 02 21:29:57.997394 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.997367    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:29:57.997539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:57.997467    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:29:58.003565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.003526    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:29:58.003684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.003632    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:29:58.008767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.008743    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"]
Nov 02 21:29:58.013278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.013261    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.052708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.052674    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc785ce6-lnv7v\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:58.052886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.052730    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.052886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.052762    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.052886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.052788    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.052886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.052821    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.052886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.052844    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.053216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.052886    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc785ce6-lnv7v\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:58.053216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.052934    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.053216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.053045    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.053216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.053127    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kctc\" (UniqueName: \"kubernetes.io/projected/0b0c3f31-1d14-4535-bfc5-4af2e0233682-kube-api-access-8kctc\") pod \"maas-api-revoke-sub-bc785ce6-lnv7v\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:58.053216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.053169    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5r8\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-kube-api-access-2f5r8\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.053216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.053214    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.053663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.053638    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc785ce6-lnv7v\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:58.057362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.056295    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc785ce6-lnv7v\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:58.072672 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.072642    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"]
Nov 02 21:29:58.104362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.104336    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-70efbbab-openshift-default-dockercfg-m2wbl\""
Nov 02 21:29:58.116480 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.116452    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kctc\" (UniqueName: \"kubernetes.io/projected/0b0c3f31-1d14-4535-bfc5-4af2e0233682-kube-api-access-8kctc\") pod \"maas-api-revoke-sub-bc785ce6-lnv7v\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:58.153980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.153948    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154032    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5r8\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-kube-api-access-2f5r8\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154075    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154141    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154169    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154206    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154244    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154269    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154346    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154516    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154538    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.154956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.154936    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.155230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.155213    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.155318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.155261    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.156318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.156298    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.156808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.156792    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.211275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.211205    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.211593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.211276    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5r8\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-kube-api-access-2f5r8\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.288134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.287327    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:29:58.349134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.330514    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:29:58.871728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.871650    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"]
Nov 02 21:29:58.907173 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:58.907146    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0c3f31_1d14_4535_bfc5_4af2e0233682.slice/crio-e0b0ca1007940e45a6837895533f2185b1df68691392c69e39ac9edf777a2ca8 WatchSource:0}: Error finding container e0b0ca1007940e45a6837895533f2185b1df68691392c69e39ac9edf777a2ca8: Status 404 returned error can't find the container with id e0b0ca1007940e45a6837895533f2185b1df68691392c69e39ac9edf777a2ca8
Nov 02 21:29:58.920955 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:29:58.920919    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae9bbaf_405b_43a8_b353_73a2265a1ad3.slice/crio-ab5e46bca66d274cc1d10573dd7d878eb8f59927f30f62bea1c668081b9b43bd WatchSource:0}: Error finding container ab5e46bca66d274cc1d10573dd7d878eb8f59927f30f62bea1c668081b9b43bd: Status 404 returned error can't find the container with id ab5e46bca66d274cc1d10573dd7d878eb8f59927f30f62bea1c668081b9b43bd
Nov 02 21:29:58.923726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.923695    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:58.923814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.923767    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:58.923874 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.923817    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:29:58.924626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:58.924608    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"]
Nov 02 21:29:59.545816 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:59.545779    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v" event={"ID":"0b0c3f31-1d14-4535-bfc5-4af2e0233682","Type":"ContainerStarted","Data":"97ea87470bd7a38b6fdec1465ab345689a2d09cf7bfc2624dc6460bddc3a3b22"}
Nov 02 21:29:59.546262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:59.545838    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v" event={"ID":"0b0c3f31-1d14-4535-bfc5-4af2e0233682","Type":"ContainerStarted","Data":"e0b0ca1007940e45a6837895533f2185b1df68691392c69e39ac9edf777a2ca8"}
Nov 02 21:29:59.547470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:59.547445    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" event={"ID":"2ae9bbaf-405b-43a8-b353-73a2265a1ad3","Type":"ContainerStarted","Data":"ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150"}
Nov 02 21:29:59.547589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:59.547483    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" event={"ID":"2ae9bbaf-405b-43a8-b353-73a2265a1ad3","Type":"ContainerStarted","Data":"ab5e46bca66d274cc1d10573dd7d878eb8f59927f30f62bea1c668081b9b43bd"}
Nov 02 21:29:59.661525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:59.661471    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v" podStartSLOduration=2.661457101 podStartE2EDuration="2.661457101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:59.658337868 +0000 UTC m=+1286.921874184" watchObservedRunningTime="2026-11-02 21:29:59.661457101 +0000 UTC m=+1286.924993419"
Nov 02 21:29:59.776261 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:29:59.776215    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" podStartSLOduration=2.776196758 podStartE2EDuration="2.776196758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:59.773586409 +0000 UTC m=+1287.037122739" watchObservedRunningTime="2026-11-02 21:29:59.776196758 +0000 UTC m=+1287.039733074"
Nov 02 21:30:00.168794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.168760    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"]
Nov 02 21:30:00.177200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.176263    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:00.177681 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.177100    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"]
Nov 02 21:30:00.183697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.183671    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"]
Nov 02 21:30:00.184036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.183802    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:00.187864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.187847    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:00.188915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.188895    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"]
Nov 02 21:30:00.193268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.193248    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:00.238849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.238823    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqswn\" (UniqueName: \"kubernetes.io/projected/6b8c7682-bb71-415c-bed2-0df4b0d76505-kube-api-access-tqswn\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj\" (UID: \"6b8c7682-bb71-415c-bed2-0df4b0d76505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:00.238974 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.238892    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6j2p\" (UniqueName: \"kubernetes.io/projected/8797d951-1a44-4b26-ae7a-84151cd30fa5-kube-api-access-f6j2p\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck\" (UID: \"8797d951-1a44-4b26-ae7a-84151cd30fa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:00.239023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.239004    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057191c4-79fe-4587-a44d-4cea640780fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6\" (UID: \"057191c4-79fe-4587-a44d-4cea640780fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:00.239056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.239041    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnmt8\" (UniqueName: \"kubernetes.io/projected/057191c4-79fe-4587-a44d-4cea640780fa-kube-api-access-nnmt8\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6\" (UID: \"057191c4-79fe-4587-a44d-4cea640780fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:00.239104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.239063    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw\" (UID: \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:00.239198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.239178    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c7682-bb71-415c-bed2-0df4b0d76505-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj\" (UID: \"6b8c7682-bb71-415c-bed2-0df4b0d76505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:00.239239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.239210    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d951-1a44-4b26-ae7a-84151cd30fa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck\" (UID: \"8797d951-1a44-4b26-ae7a-84151cd30fa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:00.239318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.239307    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4w2\" (UniqueName: \"kubernetes.io/projected/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-kube-api-access-5d4w2\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw\" (UID: \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:00.263588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.263561    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"]
Nov 02 21:30:00.267700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.267676    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"]
Nov 02 21:30:00.314800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.314765    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"]
Nov 02 21:30:00.320192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.320162    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"]
Nov 02 21:30:00.331483 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.331454    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:30:00.337040 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.337015    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:30:00.340641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.340616    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqswn\" (UniqueName: \"kubernetes.io/projected/6b8c7682-bb71-415c-bed2-0df4b0d76505-kube-api-access-tqswn\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj\" (UID: \"6b8c7682-bb71-415c-bed2-0df4b0d76505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:00.340744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.340696    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6j2p\" (UniqueName: \"kubernetes.io/projected/8797d951-1a44-4b26-ae7a-84151cd30fa5-kube-api-access-f6j2p\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck\" (UID: \"8797d951-1a44-4b26-ae7a-84151cd30fa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:00.340805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.340754    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057191c4-79fe-4587-a44d-4cea640780fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6\" (UID: \"057191c4-79fe-4587-a44d-4cea640780fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:00.340805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.340789    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnmt8\" (UniqueName: \"kubernetes.io/projected/057191c4-79fe-4587-a44d-4cea640780fa-kube-api-access-nnmt8\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6\" (UID: \"057191c4-79fe-4587-a44d-4cea640780fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:00.340904 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.340815    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw\" (UID: \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:00.340904 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.340844    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c7682-bb71-415c-bed2-0df4b0d76505-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj\" (UID: \"6b8c7682-bb71-415c-bed2-0df4b0d76505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:00.340904 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.340868    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d951-1a44-4b26-ae7a-84151cd30fa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck\" (UID: \"8797d951-1a44-4b26-ae7a-84151cd30fa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:00.341087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.340934    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4w2\" (UniqueName: \"kubernetes.io/projected/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-kube-api-access-5d4w2\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw\" (UID: \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:00.341636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.341613    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057191c4-79fe-4587-a44d-4cea640780fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6\" (UID: \"057191c4-79fe-4587-a44d-4cea640780fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:00.341714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.341691    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw\" (UID: \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:00.341753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.341704    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c7682-bb71-415c-bed2-0df4b0d76505-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj\" (UID: \"6b8c7682-bb71-415c-bed2-0df4b0d76505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:00.341788 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.341770    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d951-1a44-4b26-ae7a-84151cd30fa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck\" (UID: \"8797d951-1a44-4b26-ae7a-84151cd30fa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:00.399314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.399289    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqswn\" (UniqueName: \"kubernetes.io/projected/6b8c7682-bb71-415c-bed2-0df4b0d76505-kube-api-access-tqswn\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj\" (UID: \"6b8c7682-bb71-415c-bed2-0df4b0d76505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:00.399613 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.399594    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnmt8\" (UniqueName: \"kubernetes.io/projected/057191c4-79fe-4587-a44d-4cea640780fa-kube-api-access-nnmt8\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6\" (UID: \"057191c4-79fe-4587-a44d-4cea640780fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:00.399818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.399798    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4w2\" (UniqueName: \"kubernetes.io/projected/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-kube-api-access-5d4w2\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw\" (UID: \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:00.400001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.399982    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6j2p\" (UniqueName: \"kubernetes.io/projected/8797d951-1a44-4b26-ae7a-84151cd30fa5-kube-api-access-f6j2p\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck\" (UID: \"8797d951-1a44-4b26-ae7a-84151cd30fa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:00.499251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.499166    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:00.509709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.509684    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:00.516800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.516771    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:00.524138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.524088    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:00.560474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.560440    4528 generic.go:356] "Generic (PLEG): container finished" podID="0b0c3f31-1d14-4535-bfc5-4af2e0233682" containerID="97ea87470bd7a38b6fdec1465ab345689a2d09cf7bfc2624dc6460bddc3a3b22" exitCode=0
Nov 02 21:30:00.562629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.562149    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v" event={"ID":"0b0c3f31-1d14-4535-bfc5-4af2e0233682","Type":"ContainerDied","Data":"97ea87470bd7a38b6fdec1465ab345689a2d09cf7bfc2624dc6460bddc3a3b22"}
Nov 02 21:30:00.562629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.562248    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:30:00.563602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.563580    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:30:00.911457 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:00.911424    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8c7682_bb71_415c_bed2_0df4b0d76505.slice/crio-1d7a06edea6898b3adbb76269a4da7c0a95ddfb61250f66837214dd51c2be40b WatchSource:0}: Error finding container 1d7a06edea6898b3adbb76269a4da7c0a95ddfb61250f66837214dd51c2be40b: Status 404 returned error can't find the container with id 1d7a06edea6898b3adbb76269a4da7c0a95ddfb61250f66837214dd51c2be40b
Nov 02 21:30:00.915498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.915475    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"]
Nov 02 21:30:00.941192 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:00.941166    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8797d951_1a44_4b26_ae7a_84151cd30fa5.slice/crio-3650c1d9913a2d3ae9cc14e536618ee98be4d6dec4e7d4b452bc62fd9d8754fd WatchSource:0}: Error finding container 3650c1d9913a2d3ae9cc14e536618ee98be4d6dec4e7d4b452bc62fd9d8754fd: Status 404 returned error can't find the container with id 3650c1d9913a2d3ae9cc14e536618ee98be4d6dec4e7d4b452bc62fd9d8754fd
Nov 02 21:30:00.942986 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.942961    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"]
Nov 02 21:30:00.947474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.947444    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"]
Nov 02 21:30:00.950384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:00.950364    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"]
Nov 02 21:30:01.003824 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:01.003787    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057191c4_79fe_4587_a44d_4cea640780fa.slice/crio-8f4a9faf4ed2c59cfa5205fd488919e347a3b3f7a2c549185d2c702e69ce1c7f WatchSource:0}: Error finding container 8f4a9faf4ed2c59cfa5205fd488919e347a3b3f7a2c549185d2c702e69ce1c7f: Status 404 returned error can't find the container with id 8f4a9faf4ed2c59cfa5205fd488919e347a3b3f7a2c549185d2c702e69ce1c7f
Nov 02 21:30:01.568779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.568744    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" event={"ID":"8797d951-1a44-4b26-ae7a-84151cd30fa5","Type":"ContainerStarted","Data":"083e83fdc32ccce3fcd28cabf0e5dbb226f0526a6a32f3601347a0b5bc7e3c37"}
Nov 02 21:30:01.569307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.568804    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" event={"ID":"8797d951-1a44-4b26-ae7a-84151cd30fa5","Type":"ContainerStarted","Data":"3650c1d9913a2d3ae9cc14e536618ee98be4d6dec4e7d4b452bc62fd9d8754fd"}
Nov 02 21:30:01.570642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.570617    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" event={"ID":"a044ae68-84f0-4d5c-9ba9-0cfe2999d332","Type":"ContainerStarted","Data":"2fc71e856093771bcb517b61abe902db948bf6098dab8e4be48aa65211aaeb63"}
Nov 02 21:30:01.570776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.570662    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" event={"ID":"a044ae68-84f0-4d5c-9ba9-0cfe2999d332","Type":"ContainerStarted","Data":"4a4395f82f2ec64a3a9d1f7bef2bf95adac11d903e1c4304706193f539ce81fb"}
Nov 02 21:30:01.572255 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.572234    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" event={"ID":"057191c4-79fe-4587-a44d-4cea640780fa","Type":"ContainerStarted","Data":"c376e1aaab7f405d102b89536cbcadae8bbe92b36aa290b2eb66d64ebcf46def"}
Nov 02 21:30:01.572329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.572275    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" event={"ID":"057191c4-79fe-4587-a44d-4cea640780fa","Type":"ContainerStarted","Data":"8f4a9faf4ed2c59cfa5205fd488919e347a3b3f7a2c549185d2c702e69ce1c7f"}
Nov 02 21:30:01.573688 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.573667    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" event={"ID":"6b8c7682-bb71-415c-bed2-0df4b0d76505","Type":"ContainerStarted","Data":"34f5e8630da2890bee339c7db80a736fd631317f1e3830b6757809d88f59db53"}
Nov 02 21:30:01.573771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.573708    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" event={"ID":"6b8c7682-bb71-415c-bed2-0df4b0d76505","Type":"ContainerStarted","Data":"1d7a06edea6898b3adbb76269a4da7c0a95ddfb61250f66837214dd51c2be40b"}
Nov 02 21:30:01.682504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.682456    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" podStartSLOduration=1.682440538 podStartE2EDuration="1.682440538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:01.681999318 +0000 UTC m=+1288.945535620" watchObservedRunningTime="2026-11-02 21:30:01.682440538 +0000 UTC m=+1288.945976855"
Nov 02 21:30:01.797493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.797440    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" podStartSLOduration=1.797425317 podStartE2EDuration="1.797425317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:01.79521092 +0000 UTC m=+1289.058747238" watchObservedRunningTime="2026-11-02 21:30:01.797425317 +0000 UTC m=+1289.060961634"
Nov 02 21:30:01.838429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.838405    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:30:01.861900 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.861879    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0c3f31-1d14-4535-bfc5-4af2e0233682-kube-api-access-8kctc\" (UniqueName: \"kubernetes.io/projected/0b0c3f31-1d14-4535-bfc5-4af2e0233682-kube-api-access-8kctc\") pod \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") "
Nov 02 21:30:01.862011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.861999    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-xks-ca\") pod \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") "
Nov 02 21:30:01.862054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.862025    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-openshift-ca\") pod \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\" (UID: \"0b0c3f31-1d14-4535-bfc5-4af2e0233682\") "
Nov 02 21:30:01.862465 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.862435    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-openshift-ca" pod "0b0c3f31-1d14-4535-bfc5-4af2e0233682" (UID: "0b0c3f31-1d14-4535-bfc5-4af2e0233682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:30:01.864001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.863979    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-xks-ca" pod "0b0c3f31-1d14-4535-bfc5-4af2e0233682" (UID: "0b0c3f31-1d14-4535-bfc5-4af2e0233682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:30:01.864091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.864004    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c3f31-1d14-4535-bfc5-4af2e0233682-kube-api-access-8kctc" pod "0b0c3f31-1d14-4535-bfc5-4af2e0233682" (UID: "0b0c3f31-1d14-4535-bfc5-4af2e0233682"). InnerVolumeSpecName "kube-api-access-8kctc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:30:01.909151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.908868    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" podStartSLOduration=1.9088507479999999 podStartE2EDuration="1.908850748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:01.907664624 +0000 UTC m=+1289.171200940" watchObservedRunningTime="2026-11-02 21:30:01.908850748 +0000 UTC m=+1289.172387065"
Nov 02 21:30:01.963364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.963338    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:01.963470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.963368    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c3f31-1d14-4535-bfc5-4af2e0233682-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:01.963470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:01.963382    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kctc\" (UniqueName: \"kubernetes.io/projected/0b0c3f31-1d14-4535-bfc5-4af2e0233682-kube-api-access-8kctc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:02.023124 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.023051    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" podStartSLOduration=2.023031508 podStartE2EDuration="2.023031508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:02.020190892 +0000 UTC m=+1289.283727219" watchObservedRunningTime="2026-11-02 21:30:02.023031508 +0000 UTC m=+1289.286567827"
Nov 02 21:30:02.582658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.582620    4528 generic.go:356] "Generic (PLEG): container finished" podID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerID="083e83fdc32ccce3fcd28cabf0e5dbb226f0526a6a32f3601347a0b5bc7e3c37" exitCode=60
Nov 02 21:30:02.583090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.582762    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" event={"ID":"8797d951-1a44-4b26-ae7a-84151cd30fa5","Type":"ContainerDied","Data":"083e83fdc32ccce3fcd28cabf0e5dbb226f0526a6a32f3601347a0b5bc7e3c37"}
Nov 02 21:30:02.583390 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.583373    4528 scope.go:122] "RemoveContainer" containerID="083e83fdc32ccce3fcd28cabf0e5dbb226f0526a6a32f3601347a0b5bc7e3c37"
Nov 02 21:30:02.584376 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.584356    4528 generic.go:356] "Generic (PLEG): container finished" podID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerID="2fc71e856093771bcb517b61abe902db948bf6098dab8e4be48aa65211aaeb63" exitCode=60
Nov 02 21:30:02.584475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.584440    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" event={"ID":"a044ae68-84f0-4d5c-9ba9-0cfe2999d332","Type":"ContainerDied","Data":"2fc71e856093771bcb517b61abe902db948bf6098dab8e4be48aa65211aaeb63"}
Nov 02 21:30:02.584656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.584638    4528 scope.go:122] "RemoveContainer" containerID="2fc71e856093771bcb517b61abe902db948bf6098dab8e4be48aa65211aaeb63"
Nov 02 21:30:02.586357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.586334    4528 generic.go:356] "Generic (PLEG): container finished" podID="057191c4-79fe-4587-a44d-4cea640780fa" containerID="c376e1aaab7f405d102b89536cbcadae8bbe92b36aa290b2eb66d64ebcf46def" exitCode=60
Nov 02 21:30:02.586459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.586393    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" event={"ID":"057191c4-79fe-4587-a44d-4cea640780fa","Type":"ContainerDied","Data":"c376e1aaab7f405d102b89536cbcadae8bbe92b36aa290b2eb66d64ebcf46def"}
Nov 02 21:30:02.586768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.586592    4528 scope.go:122] "RemoveContainer" containerID="c376e1aaab7f405d102b89536cbcadae8bbe92b36aa290b2eb66d64ebcf46def"
Nov 02 21:30:02.588181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.588163    4528 generic.go:356] "Generic (PLEG): container finished" podID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerID="34f5e8630da2890bee339c7db80a736fd631317f1e3830b6757809d88f59db53" exitCode=60
Nov 02 21:30:02.588267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.588239    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" event={"ID":"6b8c7682-bb71-415c-bed2-0df4b0d76505","Type":"ContainerDied","Data":"34f5e8630da2890bee339c7db80a736fd631317f1e3830b6757809d88f59db53"}
Nov 02 21:30:02.588468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.588450    4528 scope.go:122] "RemoveContainer" containerID="34f5e8630da2890bee339c7db80a736fd631317f1e3830b6757809d88f59db53"
Nov 02 21:30:02.589942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.589925    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"
Nov 02 21:30:02.590045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.589922    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v" event={"ID":"0b0c3f31-1d14-4535-bfc5-4af2e0233682","Type":"ContainerDied","Data":"e0b0ca1007940e45a6837895533f2185b1df68691392c69e39ac9edf777a2ca8"}
Nov 02 21:30:02.590100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:02.590072    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0b0ca1007940e45a6837895533f2185b1df68691392c69e39ac9edf777a2ca8"
Nov 02 21:30:03.599043 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:03.599009    4528 generic.go:356] "Generic (PLEG): container finished" podID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerID="dac7075ee42e86c16c0e72f68b60e211cb2ee37a42a699dfc91247fad503f13b" exitCode=60
Nov 02 21:30:03.599446 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:03.599088    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" event={"ID":"a044ae68-84f0-4d5c-9ba9-0cfe2999d332","Type":"ContainerDied","Data":"dac7075ee42e86c16c0e72f68b60e211cb2ee37a42a699dfc91247fad503f13b"}
Nov 02 21:30:03.599446 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:03.599208    4528 scope.go:122] "RemoveContainer" containerID="2fc71e856093771bcb517b61abe902db948bf6098dab8e4be48aa65211aaeb63"
Nov 02 21:30:03.599547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:03.599449    4528 scope.go:122] "RemoveContainer" containerID="dac7075ee42e86c16c0e72f68b60e211cb2ee37a42a699dfc91247fad503f13b"
Nov 02 21:30:03.599783 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:03.599757    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw_odh-ai-gateway-infra(a044ae68-84f0-4d5c-9ba9-0cfe2999d332)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332"
Nov 02 21:30:03.601182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:03.601151    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" event={"ID":"057191c4-79fe-4587-a44d-4cea640780fa","Type":"ContainerStarted","Data":"380ed1165237021dc7c8de1b70177ff78d3d7de4c11e083bbf85b5cbdb3bea94"}
Nov 02 21:30:03.603026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:03.603003    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" event={"ID":"6b8c7682-bb71-415c-bed2-0df4b0d76505","Type":"ContainerStarted","Data":"e90b2e3ecbb2462073262a8a9975f81e6bdc6e4aae8893346fa93cae99d5204e"}
Nov 02 21:30:03.605469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:03.605447    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" event={"ID":"8797d951-1a44-4b26-ae7a-84151cd30fa5","Type":"ContainerStarted","Data":"da85f3c5e495b7c1624e7df1ee2bc0c56264f346445f12675350aa7a55285ee4"}
Nov 02 21:30:04.614838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.614809    4528 scope.go:122] "RemoveContainer" containerID="dac7075ee42e86c16c0e72f68b60e211cb2ee37a42a699dfc91247fad503f13b"
Nov 02 21:30:04.615288 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:04.615091    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw_odh-ai-gateway-infra(a044ae68-84f0-4d5c-9ba9-0cfe2999d332)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332"
Nov 02 21:30:04.616755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.616725    4528 generic.go:356] "Generic (PLEG): container finished" podID="057191c4-79fe-4587-a44d-4cea640780fa" containerID="380ed1165237021dc7c8de1b70177ff78d3d7de4c11e083bbf85b5cbdb3bea94" exitCode=60
Nov 02 21:30:04.616902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.616801    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" event={"ID":"057191c4-79fe-4587-a44d-4cea640780fa","Type":"ContainerDied","Data":"380ed1165237021dc7c8de1b70177ff78d3d7de4c11e083bbf85b5cbdb3bea94"}
Nov 02 21:30:04.616902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.616860    4528 scope.go:122] "RemoveContainer" containerID="c376e1aaab7f405d102b89536cbcadae8bbe92b36aa290b2eb66d64ebcf46def"
Nov 02 21:30:04.617061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.617028    4528 scope.go:122] "RemoveContainer" containerID="380ed1165237021dc7c8de1b70177ff78d3d7de4c11e083bbf85b5cbdb3bea94"
Nov 02 21:30:04.617300 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:04.617273    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6_odh-ai-gateway-infra(057191c4-79fe-4587-a44d-4cea640780fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" podUID="057191c4-79fe-4587-a44d-4cea640780fa"
Nov 02 21:30:04.619056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.619034    4528 generic.go:356] "Generic (PLEG): container finished" podID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerID="e90b2e3ecbb2462073262a8a9975f81e6bdc6e4aae8893346fa93cae99d5204e" exitCode=60
Nov 02 21:30:04.619155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.619134    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" event={"ID":"6b8c7682-bb71-415c-bed2-0df4b0d76505","Type":"ContainerDied","Data":"e90b2e3ecbb2462073262a8a9975f81e6bdc6e4aae8893346fa93cae99d5204e"}
Nov 02 21:30:04.619319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.619303    4528 scope.go:122] "RemoveContainer" containerID="e90b2e3ecbb2462073262a8a9975f81e6bdc6e4aae8893346fa93cae99d5204e"
Nov 02 21:30:04.619486 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:04.619470    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj_odh-ai-gateway-infra(6b8c7682-bb71-415c-bed2-0df4b0d76505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505"
Nov 02 21:30:04.620948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.620921    4528 generic.go:356] "Generic (PLEG): container finished" podID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerID="da85f3c5e495b7c1624e7df1ee2bc0c56264f346445f12675350aa7a55285ee4" exitCode=60
Nov 02 21:30:04.621064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.621004    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" event={"ID":"8797d951-1a44-4b26-ae7a-84151cd30fa5","Type":"ContainerDied","Data":"da85f3c5e495b7c1624e7df1ee2bc0c56264f346445f12675350aa7a55285ee4"}
Nov 02 21:30:04.621195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.621181    4528 scope.go:122] "RemoveContainer" containerID="da85f3c5e495b7c1624e7df1ee2bc0c56264f346445f12675350aa7a55285ee4"
Nov 02 21:30:04.621410 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:04.621390    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck_odh-ai-gateway-infra(8797d951-1a44-4b26-ae7a-84151cd30fa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5"
Nov 02 21:30:04.630915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.630897    4528 scope.go:122] "RemoveContainer" containerID="34f5e8630da2890bee339c7db80a736fd631317f1e3830b6757809d88f59db53"
Nov 02 21:30:04.643232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:04.643213    4528 scope.go:122] "RemoveContainer" containerID="083e83fdc32ccce3fcd28cabf0e5dbb226f0526a6a32f3601347a0b5bc7e3c37"
Nov 02 21:30:05.629073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:05.629037    4528 scope.go:122] "RemoveContainer" containerID="380ed1165237021dc7c8de1b70177ff78d3d7de4c11e083bbf85b5cbdb3bea94"
Nov 02 21:30:05.629550 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:05.629368    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6_odh-ai-gateway-infra(057191c4-79fe-4587-a44d-4cea640780fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" podUID="057191c4-79fe-4587-a44d-4cea640780fa"
Nov 02 21:30:05.630551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:05.630530    4528 scope.go:122] "RemoveContainer" containerID="e90b2e3ecbb2462073262a8a9975f81e6bdc6e4aae8893346fa93cae99d5204e"
Nov 02 21:30:05.630741 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:05.630724    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj_odh-ai-gateway-infra(6b8c7682-bb71-415c-bed2-0df4b0d76505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505"
Nov 02 21:30:05.632133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:05.632102    4528 scope.go:122] "RemoveContainer" containerID="da85f3c5e495b7c1624e7df1ee2bc0c56264f346445f12675350aa7a55285ee4"
Nov 02 21:30:05.632314 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:05.632296    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck_odh-ai-gateway-infra(8797d951-1a44-4b26-ae7a-84151cd30fa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5"
Nov 02 21:30:06.191584 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.191548    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"]
Nov 02 21:30:06.192382 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.192365    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0b0c3f31-1d14-4535-bfc5-4af2e0233682" containerName="revoke-keys"
Nov 02 21:30:06.192449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.192383    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0c3f31-1d14-4535-bfc5-4af2e0233682" containerName="revoke-keys"
Nov 02 21:30:06.192542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.192531    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0c3f31-1d14-4535-bfc5-4af2e0233682" containerName="revoke-keys"
Nov 02 21:30:06.197696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.197675    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:06.214624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.214601    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv\" (UID: \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:06.214708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.214678    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8f5w\" (UniqueName: \"kubernetes.io/projected/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-kube-api-access-n8f5w\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv\" (UID: \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:06.246162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.246132    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"]
Nov 02 21:30:06.265207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.263885    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-70efbbab\""
Nov 02 21:30:06.265207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.264609    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-70efbbab-dockercfg-89q4n\""
Nov 02 21:30:06.317018 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.316957    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv\" (UID: \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:06.356918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.356881    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8f5w\" (UniqueName: \"kubernetes.io/projected/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-kube-api-access-n8f5w\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv\" (UID: \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:06.357706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.352694    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv\" (UID: \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:06.379207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.369538    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"]
Nov 02 21:30:06.403439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.403408    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:06.422350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.422314    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8f5w\" (UniqueName: \"kubernetes.io/projected/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-kube-api-access-n8f5w\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv\" (UID: \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:06.425761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.425733    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"]
Nov 02 21:30:06.463294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.458887    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f5dff-d162-4d80-b70c-04f273bcee9b-plugins-config-volume\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-vs5c9\" (UID: \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:06.463294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.458995    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7kt\" (UniqueName: \"kubernetes.io/projected/4a1f5dff-d162-4d80-b70c-04f273bcee9b-kube-api-access-4p7kt\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-vs5c9\" (UID: \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:06.512008 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.511972    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:06.560911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.560872    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f5dff-d162-4d80-b70c-04f273bcee9b-plugins-config-volume\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-vs5c9\" (UID: \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:06.561185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.560993    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7kt\" (UniqueName: \"kubernetes.io/projected/4a1f5dff-d162-4d80-b70c-04f273bcee9b-kube-api-access-4p7kt\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-vs5c9\" (UID: \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:06.561856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.561801    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f5dff-d162-4d80-b70c-04f273bcee9b-plugins-config-volume\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-vs5c9\" (UID: \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:06.618554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.618518    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7kt\" (UniqueName: \"kubernetes.io/projected/4a1f5dff-d162-4d80-b70c-04f273bcee9b-kube-api-access-4p7kt\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-vs5c9\" (UID: \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:06.776182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.776152    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:06.882222 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:06.882174    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bfb456_7eb0_4fb0_8a1f_51fcfc7294c2.slice/crio-8b228d25edd7fe81ed9bda0bc10f0f7e0bfd405fb9a41a6f745a96ed275a0c28 WatchSource:0}: Error finding container 8b228d25edd7fe81ed9bda0bc10f0f7e0bfd405fb9a41a6f745a96ed275a0c28: Status 404 returned error can't find the container with id 8b228d25edd7fe81ed9bda0bc10f0f7e0bfd405fb9a41a6f745a96ed275a0c28
Nov 02 21:30:06.884081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:06.884054    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"]
Nov 02 21:30:07.144912 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:07.144882    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1f5dff_d162_4d80_b70c_04f273bcee9b.slice/crio-dbc07437bc6061bde14e6675d3e9355a57b650a5f00192618342a4e918263bcc WatchSource:0}: Error finding container dbc07437bc6061bde14e6675d3e9355a57b650a5f00192618342a4e918263bcc: Status 404 returned error can't find the container with id dbc07437bc6061bde14e6675d3e9355a57b650a5f00192618342a4e918263bcc
Nov 02 21:30:07.148001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:07.147896    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"]
Nov 02 21:30:07.647380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:07.647350    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9" event={"ID":"4a1f5dff-d162-4d80-b70c-04f273bcee9b","Type":"ContainerStarted","Data":"a45b29ac0aca8f5a1057dba8e8dfdef6722adee6fee4dee75e8add3187a39530"}
Nov 02 21:30:07.647583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:07.647401    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9" event={"ID":"4a1f5dff-d162-4d80-b70c-04f273bcee9b","Type":"ContainerStarted","Data":"dbc07437bc6061bde14e6675d3e9355a57b650a5f00192618342a4e918263bcc"}
Nov 02 21:30:07.647583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:07.647549    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:07.649059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:07.649032    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv" event={"ID":"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2","Type":"ContainerStarted","Data":"55cec3fe2ba6388358de158acfa1a72fca6f67c304cb3bbef21646af54ac63f2"}
Nov 02 21:30:07.649165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:07.649081    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv" event={"ID":"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2","Type":"ContainerStarted","Data":"8b228d25edd7fe81ed9bda0bc10f0f7e0bfd405fb9a41a6f745a96ed275a0c28"}
Nov 02 21:30:07.649165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:07.649155    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:07.763649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:07.763582    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9" podStartSLOduration=1.763566044 podStartE2EDuration="1.763566044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:07.760782982 +0000 UTC m=+1295.024319299" watchObservedRunningTime="2026-11-02 21:30:07.763566044 +0000 UTC m=+1295.027102352"
Nov 02 21:30:07.896134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:07.895838    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv" podStartSLOduration=1.895815719 podStartE2EDuration="1.895815719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:07.873314144 +0000 UTC m=+1295.136850465" watchObservedRunningTime="2026-11-02 21:30:07.895815719 +0000 UTC m=+1295.159352050"
Nov 02 21:30:11.562168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.562132    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"]
Nov 02 21:30:11.566557 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.566537    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.619177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.619144    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:30:11.677200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.677155    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2gh\" (UniqueName: \"kubernetes.io/projected/9d0d93c9-4b6d-4e98-938a-75b595a149a9-kube-api-access-kt2gh\") pod \"maas-api-revoke-sub-3298cc19-27swk\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.677587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.677561    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3298cc19-27swk\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.677838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.677819    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3298cc19-27swk\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.699208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.699164    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"]
Nov 02 21:30:11.779430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.779390    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2gh\" (UniqueName: \"kubernetes.io/projected/9d0d93c9-4b6d-4e98-938a-75b595a149a9-kube-api-access-kt2gh\") pod \"maas-api-revoke-sub-3298cc19-27swk\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.779775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.779752    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3298cc19-27swk\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.779995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.779975    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3298cc19-27swk\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.798214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.795279    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3298cc19-27swk\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.802451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.802408    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3298cc19-27swk\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.852057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.851726    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2gh\" (UniqueName: \"kubernetes.io/projected/9d0d93c9-4b6d-4e98-938a-75b595a149a9-kube-api-access-kt2gh\") pod \"maas-api-revoke-sub-3298cc19-27swk\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:11.886767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:11.886738    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:12.258075 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:12.258048    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0d93c9_4b6d_4e98_938a_75b595a149a9.slice/crio-ab4248077e2d3389e18383b2d726e5377adad0b98be3f3b892e1088f230a0a19 WatchSource:0}: Error finding container ab4248077e2d3389e18383b2d726e5377adad0b98be3f3b892e1088f230a0a19: Status 404 returned error can't find the container with id ab4248077e2d3389e18383b2d726e5377adad0b98be3f3b892e1088f230a0a19
Nov 02 21:30:12.262195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:12.262173    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"]
Nov 02 21:30:12.745720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:12.745680    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk" event={"ID":"9d0d93c9-4b6d-4e98-938a-75b595a149a9","Type":"ContainerStarted","Data":"a66ca22ee8656b6758fdf10a1b7cff46a3d69edd9077701366ac0e77a30299a6"}
Nov 02 21:30:12.746142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:12.745747    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk" event={"ID":"9d0d93c9-4b6d-4e98-938a-75b595a149a9","Type":"ContainerStarted","Data":"ab4248077e2d3389e18383b2d726e5377adad0b98be3f3b892e1088f230a0a19"}
Nov 02 21:30:12.866610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:12.866507    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk" podStartSLOduration=1.866492665 podStartE2EDuration="1.866492665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:12.863759692 +0000 UTC m=+1300.127296009" watchObservedRunningTime="2026-11-02 21:30:12.866492665 +0000 UTC m=+1300.130029026"
Nov 02 21:30:13.756337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:13.756302    4528 generic.go:356] "Generic (PLEG): container finished" podID="9d0d93c9-4b6d-4e98-938a-75b595a149a9" containerID="a66ca22ee8656b6758fdf10a1b7cff46a3d69edd9077701366ac0e77a30299a6" exitCode=0
Nov 02 21:30:13.756736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:13.756382    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk" event={"ID":"9d0d93c9-4b6d-4e98-938a-75b595a149a9","Type":"ContainerDied","Data":"a66ca22ee8656b6758fdf10a1b7cff46a3d69edd9077701366ac0e77a30299a6"}
Nov 02 21:30:14.113737 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:14.113702    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:30:14.905423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:14.905395    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:14.906104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:14.906086    4528 scope.go:122] "RemoveContainer" containerID="380ed1165237021dc7c8de1b70177ff78d3d7de4c11e083bbf85b5cbdb3bea94"
Nov 02 21:30:15.017711 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.017643    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-xks-ca\") pod \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") "
Nov 02 21:30:15.017847 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.017782    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-openshift-ca\") pod \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") "
Nov 02 21:30:15.017928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.017880    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0d93c9-4b6d-4e98-938a-75b595a149a9-kube-api-access-kt2gh\" (UniqueName: \"kubernetes.io/projected/9d0d93c9-4b6d-4e98-938a-75b595a149a9-kube-api-access-kt2gh\") pod \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\" (UID: \"9d0d93c9-4b6d-4e98-938a-75b595a149a9\") "
Nov 02 21:30:15.018220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.018175    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-openshift-ca" pod "9d0d93c9-4b6d-4e98-938a-75b595a149a9" (UID: "9d0d93c9-4b6d-4e98-938a-75b595a149a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:30:15.019328 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.019302    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:15.023776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.023752    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-xks-ca" pod "9d0d93c9-4b6d-4e98-938a-75b595a149a9" (UID: "9d0d93c9-4b6d-4e98-938a-75b595a149a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:30:15.023860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.023795    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0d93c9-4b6d-4e98-938a-75b595a149a9-kube-api-access-kt2gh" pod "9d0d93c9-4b6d-4e98-938a-75b595a149a9" (UID: "9d0d93c9-4b6d-4e98-938a-75b595a149a9"). InnerVolumeSpecName "kube-api-access-kt2gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:30:15.120420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.120392    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0d93c9-4b6d-4e98-938a-75b595a149a9-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:15.120420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.120421    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2gh\" (UniqueName: \"kubernetes.io/projected/9d0d93c9-4b6d-4e98-938a-75b595a149a9-kube-api-access-kt2gh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:15.774528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.774501    4528 generic.go:356] "Generic (PLEG): container finished" podID="057191c4-79fe-4587-a44d-4cea640780fa" containerID="83812cb88004e047f9d37a385245fc77c084f1e0f0db8961c19c3505d6c62396" exitCode=60
Nov 02 21:30:15.774679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.774575    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" event={"ID":"057191c4-79fe-4587-a44d-4cea640780fa","Type":"ContainerDied","Data":"83812cb88004e047f9d37a385245fc77c084f1e0f0db8961c19c3505d6c62396"}
Nov 02 21:30:15.774679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.774661    4528 scope.go:122] "RemoveContainer" containerID="380ed1165237021dc7c8de1b70177ff78d3d7de4c11e083bbf85b5cbdb3bea94"
Nov 02 21:30:15.774987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.774973    4528 scope.go:122] "RemoveContainer" containerID="83812cb88004e047f9d37a385245fc77c084f1e0f0db8961c19c3505d6c62396"
Nov 02 21:30:15.775325 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:15.775304    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6_odh-ai-gateway-infra(057191c4-79fe-4587-a44d-4cea640780fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" podUID="057191c4-79fe-4587-a44d-4cea640780fa"
Nov 02 21:30:15.776213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.776193    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"
Nov 02 21:30:15.776300 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.776195    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk" event={"ID":"9d0d93c9-4b6d-4e98-938a-75b595a149a9","Type":"ContainerDied","Data":"ab4248077e2d3389e18383b2d726e5377adad0b98be3f3b892e1088f230a0a19"}
Nov 02 21:30:15.776348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.776302    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab4248077e2d3389e18383b2d726e5377adad0b98be3f3b892e1088f230a0a19"
Nov 02 21:30:15.906522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:15.906498    4528 scope.go:122] "RemoveContainer" containerID="e90b2e3ecbb2462073262a8a9975f81e6bdc6e4aae8893346fa93cae99d5204e"
Nov 02 21:30:16.785514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:16.785485    4528 generic.go:356] "Generic (PLEG): container finished" podID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerID="33a03add3ced01b9047d03f753982f876c1e8a75dda5faac49c4cb21a151313d" exitCode=60
Nov 02 21:30:16.785684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:16.785565    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" event={"ID":"6b8c7682-bb71-415c-bed2-0df4b0d76505","Type":"ContainerDied","Data":"33a03add3ced01b9047d03f753982f876c1e8a75dda5faac49c4cb21a151313d"}
Nov 02 21:30:16.785684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:16.785628    4528 scope.go:122] "RemoveContainer" containerID="e90b2e3ecbb2462073262a8a9975f81e6bdc6e4aae8893346fa93cae99d5204e"
Nov 02 21:30:16.785999 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:16.785982    4528 scope.go:122] "RemoveContainer" containerID="33a03add3ced01b9047d03f753982f876c1e8a75dda5faac49c4cb21a151313d"
Nov 02 21:30:16.786321 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:16.786303    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj_odh-ai-gateway-infra(6b8c7682-bb71-415c-bed2-0df4b0d76505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505"
Nov 02 21:30:16.899329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:16.899249    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"]
Nov 02 21:30:16.906440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:16.906416    4528 scope.go:122] "RemoveContainer" containerID="da85f3c5e495b7c1624e7df1ee2bc0c56264f346445f12675350aa7a55285ee4"
Nov 02 21:30:17.130997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.130968    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:17.244139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.244093    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057191c4-79fe-4587-a44d-4cea640780fa-kube-api-access-nnmt8\" (UniqueName: \"kubernetes.io/projected/057191c4-79fe-4587-a44d-4cea640780fa-kube-api-access-nnmt8\") pod \"057191c4-79fe-4587-a44d-4cea640780fa\" (UID: \"057191c4-79fe-4587-a44d-4cea640780fa\") "
Nov 02 21:30:17.244414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.244392    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057191c4-79fe-4587-a44d-4cea640780fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057191c4-79fe-4587-a44d-4cea640780fa-maas-api-service-ca\") pod \"057191c4-79fe-4587-a44d-4cea640780fa\" (UID: \"057191c4-79fe-4587-a44d-4cea640780fa\") "
Nov 02 21:30:17.244758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.244735    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057191c4-79fe-4587-a44d-4cea640780fa-maas-api-service-ca" pod "057191c4-79fe-4587-a44d-4cea640780fa" (UID: "057191c4-79fe-4587-a44d-4cea640780fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:30:17.245073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.245052    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057191c4-79fe-4587-a44d-4cea640780fa-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:17.247224 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.247191    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057191c4-79fe-4587-a44d-4cea640780fa-kube-api-access-nnmt8" pod "057191c4-79fe-4587-a44d-4cea640780fa" (UID: "057191c4-79fe-4587-a44d-4cea640780fa"). InnerVolumeSpecName "kube-api-access-nnmt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:30:17.345885 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.345859    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnmt8\" (UniqueName: \"kubernetes.io/projected/057191c4-79fe-4587-a44d-4cea640780fa-kube-api-access-nnmt8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:17.701222 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.701192    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:30:17.794593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.794559    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" event={"ID":"8797d951-1a44-4b26-ae7a-84151cd30fa5","Type":"ContainerStarted","Data":"e5d6488a406d8480c51535f4ab32f9603ab38336bd07e88fb413cc7cb321d4c7"}
Nov 02 21:30:17.795949 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.795920    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"
Nov 02 21:30:17.796087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.795923    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6" event={"ID":"057191c4-79fe-4587-a44d-4cea640780fa","Type":"ContainerDied","Data":"8f4a9faf4ed2c59cfa5205fd488919e347a3b3f7a2c549185d2c702e69ce1c7f"}
Nov 02 21:30:17.796087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.796051    4528 scope.go:122] "RemoveContainer" containerID="83812cb88004e047f9d37a385245fc77c084f1e0f0db8961c19c3505d6c62396"
Nov 02 21:30:17.906439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.906406    4528 scope.go:122] "RemoveContainer" containerID="dac7075ee42e86c16c0e72f68b60e211cb2ee37a42a699dfc91247fad503f13b"
Nov 02 21:30:17.916135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:17.916099    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"]
Nov 02 21:30:18.044968 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.044664    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"]
Nov 02 21:30:18.057491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.057459    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894250-vv6t6"]
Nov 02 21:30:18.123138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.123037    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:18.265027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.264996    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8c7682-bb71-415c-bed2-0df4b0d76505-kube-api-access-tqswn\" (UniqueName: \"kubernetes.io/projected/6b8c7682-bb71-415c-bed2-0df4b0d76505-kube-api-access-tqswn\") pod \"6b8c7682-bb71-415c-bed2-0df4b0d76505\" (UID: \"6b8c7682-bb71-415c-bed2-0df4b0d76505\") "
Nov 02 21:30:18.267980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.267956    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8c7682-bb71-415c-bed2-0df4b0d76505-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c7682-bb71-415c-bed2-0df4b0d76505-maas-api-service-ca\") pod \"6b8c7682-bb71-415c-bed2-0df4b0d76505\" (UID: \"6b8c7682-bb71-415c-bed2-0df4b0d76505\") "
Nov 02 21:30:18.269231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.269204    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8c7682-bb71-415c-bed2-0df4b0d76505-maas-api-service-ca" pod "6b8c7682-bb71-415c-bed2-0df4b0d76505" (UID: "6b8c7682-bb71-415c-bed2-0df4b0d76505"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:30:18.279661 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.279621    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8c7682-bb71-415c-bed2-0df4b0d76505-kube-api-access-tqswn" pod "6b8c7682-bb71-415c-bed2-0df4b0d76505" (UID: "6b8c7682-bb71-415c-bed2-0df4b0d76505"). InnerVolumeSpecName "kube-api-access-tqswn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:30:18.412177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.408706    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c7682-bb71-415c-bed2-0df4b0d76505-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:18.412177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.408744    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqswn\" (UniqueName: \"kubernetes.io/projected/6b8c7682-bb71-415c-bed2-0df4b0d76505-kube-api-access-tqswn\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:18.669736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.668415    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:30:18.683146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.676859    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:30:18.815025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.814932    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj" event={"ID":"6b8c7682-bb71-415c-bed2-0df4b0d76505","Type":"ContainerDied","Data":"1d7a06edea6898b3adbb76269a4da7c0a95ddfb61250f66837214dd51c2be40b"}
Nov 02 21:30:18.815286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.815265    4528 scope.go:122] "RemoveContainer" containerID="33a03add3ced01b9047d03f753982f876c1e8a75dda5faac49c4cb21a151313d"
Nov 02 21:30:18.815529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.815512    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"
Nov 02 21:30:18.847010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.846824    4528 generic.go:356] "Generic (PLEG): container finished" podID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerID="e5d6488a406d8480c51535f4ab32f9603ab38336bd07e88fb413cc7cb321d4c7" exitCode=60
Nov 02 21:30:18.847010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.846965    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" event={"ID":"8797d951-1a44-4b26-ae7a-84151cd30fa5","Type":"ContainerDied","Data":"e5d6488a406d8480c51535f4ab32f9603ab38336bd07e88fb413cc7cb321d4c7"}
Nov 02 21:30:18.847254 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.847026    4528 scope.go:122] "RemoveContainer" containerID="da85f3c5e495b7c1624e7df1ee2bc0c56264f346445f12675350aa7a55285ee4"
Nov 02 21:30:18.847832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.847494    4528 scope.go:122] "RemoveContainer" containerID="e5d6488a406d8480c51535f4ab32f9603ab38336bd07e88fb413cc7cb321d4c7"
Nov 02 21:30:18.847947 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:18.847908    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck_odh-ai-gateway-infra(8797d951-1a44-4b26-ae7a-84151cd30fa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5"
Nov 02 21:30:18.851704 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.851685    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" event={"ID":"a044ae68-84f0-4d5c-9ba9-0cfe2999d332","Type":"ContainerStarted","Data":"3e98f3b81597d6d9c9515ec1b05eb853ec3b8d09f9c564591db1d611d6beafff"}
Nov 02 21:30:18.921131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.920592    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057191c4-79fe-4587-a44d-4cea640780fa" path="/var/lib/kubelet/pods/057191c4-79fe-4587-a44d-4cea640780fa/volumes"
Nov 02 21:30:18.960129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:18.959195    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"]
Nov 02 21:30:19.175197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:19.173164    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"]
Nov 02 21:30:19.288150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:19.285801    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894250-cpbwj"]
Nov 02 21:30:19.929140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:19.928493    4528 generic.go:356] "Generic (PLEG): container finished" podID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerID="3e98f3b81597d6d9c9515ec1b05eb853ec3b8d09f9c564591db1d611d6beafff" exitCode=60
Nov 02 21:30:19.929140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:19.928538    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" event={"ID":"a044ae68-84f0-4d5c-9ba9-0cfe2999d332","Type":"ContainerDied","Data":"3e98f3b81597d6d9c9515ec1b05eb853ec3b8d09f9c564591db1d611d6beafff"}
Nov 02 21:30:19.929140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:19.928586    4528 scope.go:122] "RemoveContainer" containerID="dac7075ee42e86c16c0e72f68b60e211cb2ee37a42a699dfc91247fad503f13b"
Nov 02 21:30:19.929140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:19.928906    4528 scope.go:122] "RemoveContainer" containerID="3e98f3b81597d6d9c9515ec1b05eb853ec3b8d09f9c564591db1d611d6beafff"
Nov 02 21:30:19.963438 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:19.944537    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw_odh-ai-gateway-infra(a044ae68-84f0-4d5c-9ba9-0cfe2999d332)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332"
Nov 02 21:30:20.305588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:20.304493    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:20.368133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:20.367147    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"]
Nov 02 21:30:20.769134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:20.760681    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:20.789129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:20.770313    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 21:30:21.182485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:21.182442    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:21.379186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:21.333204    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" path="/var/lib/kubelet/pods/6b8c7682-bb71-415c-bed2-0df4b0d76505/volumes"
Nov 02 21:30:21.411140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:21.339325    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:21.411689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:21.341275    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:21.430299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:21.430267    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:30:21.554791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:21.554753    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8797d951-1a44-4b26-ae7a-84151cd30fa5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d951-1a44-4b26-ae7a-84151cd30fa5-maas-api-service-ca\") pod \"8797d951-1a44-4b26-ae7a-84151cd30fa5\" (UID: \"8797d951-1a44-4b26-ae7a-84151cd30fa5\") "
Nov 02 21:30:21.571210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:21.571171    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8797d951-1a44-4b26-ae7a-84151cd30fa5-kube-api-access-f6j2p\" (UniqueName: \"kubernetes.io/projected/8797d951-1a44-4b26-ae7a-84151cd30fa5-kube-api-access-f6j2p\") pod \"8797d951-1a44-4b26-ae7a-84151cd30fa5\" (UID: \"8797d951-1a44-4b26-ae7a-84151cd30fa5\") "
Nov 02 21:30:21.586376 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:21.586336    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8797d951-1a44-4b26-ae7a-84151cd30fa5-maas-api-service-ca" pod "8797d951-1a44-4b26-ae7a-84151cd30fa5" (UID: "8797d951-1a44-4b26-ae7a-84151cd30fa5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:30:21.705191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:21.680123    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d951-1a44-4b26-ae7a-84151cd30fa5-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:22.014133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.004893    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8797d951-1a44-4b26-ae7a-84151cd30fa5-kube-api-access-f6j2p" pod "8797d951-1a44-4b26-ae7a-84151cd30fa5" (UID: "8797d951-1a44-4b26-ae7a-84151cd30fa5"). InnerVolumeSpecName "kube-api-access-f6j2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:30:22.080139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.074168    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6j2p\" (UniqueName: \"kubernetes.io/projected/8797d951-1a44-4b26-ae7a-84151cd30fa5-kube-api-access-f6j2p\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:22.444132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.443379    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:22.472629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.455193    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:22.472629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.471950    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:22.538091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.538064    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"
Nov 02 21:30:22.555140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.540636    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:22.555140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.551248    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck" event={"ID":"8797d951-1a44-4b26-ae7a-84151cd30fa5","Type":"ContainerDied","Data":"3650c1d9913a2d3ae9cc14e536618ee98be4d6dec4e7d4b452bc62fd9d8754fd"}
Nov 02 21:30:22.555140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.551314    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw" event={"ID":"a044ae68-84f0-4d5c-9ba9-0cfe2999d332","Type":"ContainerDied","Data":"4a4395f82f2ec64a3a9d1f7bef2bf95adac11d903e1c4304706193f539ce81fb"}
Nov 02 21:30:22.555140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.551350    4528 scope.go:122] "RemoveContainer" containerID="e5d6488a406d8480c51535f4ab32f9603ab38336bd07e88fb413cc7cb321d4c7"
Nov 02 21:30:22.627140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.624399    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-kube-api-access-5d4w2\" (UniqueName: \"kubernetes.io/projected/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-kube-api-access-5d4w2\") pod \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\" (UID: \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\") "
Nov 02 21:30:22.627140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.624542    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-maas-api-service-ca\") pod \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\" (UID: \"a044ae68-84f0-4d5c-9ba9-0cfe2999d332\") "
Nov 02 21:30:22.627140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.626653    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-maas-api-service-ca" pod "a044ae68-84f0-4d5c-9ba9-0cfe2999d332" (UID: "a044ae68-84f0-4d5c-9ba9-0cfe2999d332"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:30:22.644144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.634654    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-kube-api-access-5d4w2" pod "a044ae68-84f0-4d5c-9ba9-0cfe2999d332" (UID: "a044ae68-84f0-4d5c-9ba9-0cfe2999d332"). InnerVolumeSpecName "kube-api-access-5d4w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:30:22.758134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.757581    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:22.758134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.757616    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4w2\" (UniqueName: \"kubernetes.io/projected/a044ae68-84f0-4d5c-9ba9-0cfe2999d332-kube-api-access-5d4w2\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:22.830157 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:22.811349    4528 scope.go:122] "RemoveContainer" containerID="3e98f3b81597d6d9c9515ec1b05eb853ec3b8d09f9c564591db1d611d6beafff"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.092717    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"]
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094161    4528 cpu_manager.go:414] "containerMap: removing container" podUID="057191c4-79fe-4587-a44d-4cea640780fa" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094184    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057191c4-79fe-4587-a44d-4cea640780fa" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094223    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094231    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094242    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9d0d93c9-4b6d-4e98-938a-75b595a149a9" containerName="revoke-keys"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094251    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0d93c9-4b6d-4e98-938a-75b595a149a9" containerName="revoke-keys"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094271    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094278    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094288    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094295    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094307    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094313    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094333    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094339    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094361    4528 cpu_manager.go:414] "containerMap: removing container" podUID="057191c4-79fe-4587-a44d-4cea640780fa" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094369    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057191c4-79fe-4587-a44d-4cea640780fa" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094380    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094387    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094397    4528 cpu_manager.go:414] "containerMap: removing container" podUID="057191c4-79fe-4587-a44d-4cea640780fa" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094404    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057191c4-79fe-4587-a44d-4cea640780fa" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094417    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094423    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094563    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094572    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094584    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094593    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094604    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="057191c4-79fe-4587-a44d-4cea640780fa" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094612    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8c7682-bb71-415c-bed2-0df4b0d76505" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094620    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="057191c4-79fe-4587-a44d-4cea640780fa" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094628    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094641    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerName="cleanup"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094649    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0d93c9-4b6d-4e98-938a-75b595a149a9" containerName="revoke-keys"
Nov 02 21:30:23.119177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.094658    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerName="cleanup"
Nov 02 21:30:23.135671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.132794    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"]
Nov 02 21:30:23.135671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.133675    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerName="cleanup"
Nov 02 21:30:23.135671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.133699    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" containerName="cleanup"
Nov 02 21:30:23.135671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.133726    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerName="cleanup"
Nov 02 21:30:23.135671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.133735    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerName="cleanup"
Nov 02 21:30:23.135671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.133896    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" containerName="cleanup"
Nov 02 21:30:23.135671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.133917    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="057191c4-79fe-4587-a44d-4cea640780fa" containerName="cleanup"
Nov 02 21:30:23.137677 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.137655    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.138210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.138175    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.172440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.170600    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"]
Nov 02 21:30:23.198738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.197832    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"]
Nov 02 21:30:23.202274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.201796    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"]
Nov 02 21:30:23.215892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.215693    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894250-m7qck"]
Nov 02 21:30:23.229560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.229516    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:30:23.314655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.314620    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.314980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.314956    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.315267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.315248    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.315534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.315517    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b62919-1afd-4451-a026-4b69da048166-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.315695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.315658    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgm46\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-kube-api-access-tgm46\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.315883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.315866    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.316023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316007    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.316120 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316041    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b62919-1afd-4451-a026-4b69da048166-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.316120 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316090    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.316258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316139    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.316258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316171    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.316258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316195    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5pw\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-kube-api-access-2h5pw\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.316258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316218    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.316258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316243    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.316531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316293    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.316531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316338    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.316531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316423    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.316531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.316497    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.432809    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b62919-1afd-4451-a026-4b69da048166-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.432865    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgm46\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-kube-api-access-tgm46\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.432897    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.432953    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433001    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b62919-1afd-4451-a026-4b69da048166-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433049    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433077    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433121    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433149    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5pw\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-kube-api-access-2h5pw\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433174    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433198    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433246    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433288    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433370    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433432    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433512    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433537    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.433590    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.435183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.434630    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b62919-1afd-4451-a026-4b69da048166-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.459131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.446755    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.459131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.446979    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.459131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.447417    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.459131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.448205    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.459131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.448437    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.459131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.448653    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.459131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.448879    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.460152 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.460091    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.460641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.460586    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.463527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.463480    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.465921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.465876    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.478917 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.478335    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b62919-1afd-4451-a026-4b69da048166-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.483679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.483515    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.491607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.491582    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.494889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.494867    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgm46\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-kube-api-access-tgm46\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.519628 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.519582    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.539682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.539647    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5pw\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-kube-api-access-2h5pw\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:23.556098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.556062    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"
Nov 02 21:30:23.724206 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.724132    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"]
Nov 02 21:30:23.730028 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.730001    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894250-bmvxw"]
Nov 02 21:30:23.772197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.772166    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:23.810351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:23.810312    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:24.290529 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:24.290493    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b62919_1afd_4451_a026_4b69da048166.slice/crio-8489ffe7b53431b618dab4daefb10fe24f94262c86fc69b9c31c4e6868702618 WatchSource:0}: Error finding container 8489ffe7b53431b618dab4daefb10fe24f94262c86fc69b9c31c4e6868702618: Status 404 returned error can't find the container with id 8489ffe7b53431b618dab4daefb10fe24f94262c86fc69b9c31c4e6868702618
Nov 02 21:30:24.296239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.295304    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"]
Nov 02 21:30:24.301427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.300733    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"]
Nov 02 21:30:24.319436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.312140    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:30:24.319436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.312236    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:30:24.319436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.312276    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:30:24.323813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.323775    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:30:24.323950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.323868    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:30:24.323950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.323909    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:30:24.567542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.567443    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" event={"ID":"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8","Type":"ContainerStarted","Data":"1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2"}
Nov 02 21:30:24.567542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.567519    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" event={"ID":"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8","Type":"ContainerStarted","Data":"1e9eb0f921299c088ac5326a1ddfa0a40e74cca52d68b213a5d57923dbc2f885"}
Nov 02 21:30:24.570225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.570194    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" event={"ID":"c9b62919-1afd-4451-a026-4b69da048166","Type":"ContainerStarted","Data":"6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a"}
Nov 02 21:30:24.570348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.570242    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" event={"ID":"c9b62919-1afd-4451-a026-4b69da048166","Type":"ContainerStarted","Data":"8489ffe7b53431b618dab4daefb10fe24f94262c86fc69b9c31c4e6868702618"}
Nov 02 21:30:24.686626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.686576    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podStartSLOduration=2.686561659 podStartE2EDuration="2.686561659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:24.685418383 +0000 UTC m=+1311.948954713" watchObservedRunningTime="2026-11-02 21:30:24.686561659 +0000 UTC m=+1311.950097975"
Nov 02 21:30:24.772861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.772827    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:24.806935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.806878    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" podStartSLOduration=2.806864235 podStartE2EDuration="2.806864235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:24.802851701 +0000 UTC m=+1312.066388041" watchObservedRunningTime="2026-11-02 21:30:24.806864235 +0000 UTC m=+1312.070400551"
Nov 02 21:30:24.810604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.810579    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:24.913793 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.913716    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8797d951-1a44-4b26-ae7a-84151cd30fa5" path="/var/lib/kubelet/pods/8797d951-1a44-4b26-ae7a-84151cd30fa5/volumes"
Nov 02 21:30:24.914150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:24.914136    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a044ae68-84f0-4d5c-9ba9-0cfe2999d332" path="/var/lib/kubelet/pods/a044ae68-84f0-4d5c-9ba9-0cfe2999d332/volumes"
Nov 02 21:30:25.774652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:25.774618    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 21:30:25.775207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:25.774676    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 21:30:25.822133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:25.818274    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 21:30:25.822133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:25.818340    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 21:30:26.782136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:26.781604    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 21:30:26.782136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:26.781703    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 21:30:26.812131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:26.811532    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 21:30:26.812131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:26.811606    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 21:30:27.772493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:27.772457    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 21:30:27.772673 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:27.772510    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 21:30:27.819133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:27.811486    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 21:30:27.819133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:27.811563    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 21:30:28.773236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:28.773204    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 21:30:28.773444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:28.773257    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 21:30:28.818141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:28.814490    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 21:30:28.818141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:28.814552    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 21:30:29.779350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:29.778835    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 21:30:29.779350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:29.778893    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 21:30:29.818134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:29.817474    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 21:30:29.818134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:29.817535    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 21:30:30.554133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.553697    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"]
Nov 02 21:30:30.560104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.560078    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.564571 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.564544    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6393175-zh5bz\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.564708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.564573    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4gq\" (UniqueName: \"kubernetes.io/projected/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-kube-api-access-2w4gq\") pod \"maas-api-revoke-sub-a6393175-zh5bz\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.564708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.564631    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6393175-zh5bz\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.609683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.609630    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"]
Nov 02 21:30:30.657160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.648639    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:30:30.657160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.656504    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:30:30.657160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.656546    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:30:30.657160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.656664    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:30:30.666103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.666072    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6393175-zh5bz\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.666103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.666103    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4gq\" (UniqueName: \"kubernetes.io/projected/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-kube-api-access-2w4gq\") pod \"maas-api-revoke-sub-a6393175-zh5bz\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.666319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.666212    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6393175-zh5bz\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.667212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.667189    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6393175-zh5bz\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.668726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.668709    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6393175-zh5bz\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.752141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.751948    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4gq\" (UniqueName: \"kubernetes.io/projected/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-kube-api-access-2w4gq\") pod \"maas-api-revoke-sub-a6393175-zh5bz\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:30.773350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.773322    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 21:30:30.773485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.773369    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 21:30:30.832183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.819459    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 21:30:30.832183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.819516    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 21:30:30.875686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:30.875653    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:31.280611 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:31.280574    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9611da1_8cb5_4d6e_a45a_b1e75c2b5e24.slice/crio-6a01c6f4f106924940c74b24ccf8a4516deecf20b5dbbc48b045b2e0118fa0a1 WatchSource:0}: Error finding container 6a01c6f4f106924940c74b24ccf8a4516deecf20b5dbbc48b045b2e0118fa0a1: Status 404 returned error can't find the container with id 6a01c6f4f106924940c74b24ccf8a4516deecf20b5dbbc48b045b2e0118fa0a1
Nov 02 21:30:31.281269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:31.281133    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"]
Nov 02 21:30:31.672626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:31.672586    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz" event={"ID":"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24","Type":"ContainerStarted","Data":"93104cb5c471788a2328e1c85dac92fd665940709c676524a346a0a1b7af9a94"}
Nov 02 21:30:31.672811 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:31.672652    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz" event={"ID":"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24","Type":"ContainerStarted","Data":"6a01c6f4f106924940c74b24ccf8a4516deecf20b5dbbc48b045b2e0118fa0a1"}
Nov 02 21:30:31.772816 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:31.772781    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 21:30:31.773220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:31.772832    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 21:30:31.787634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:31.787584    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz" podStartSLOduration=1.7875718809999999 podStartE2EDuration="1.787571881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:31.785508063 +0000 UTC m=+1319.049044382" watchObservedRunningTime="2026-11-02 21:30:31.787571881 +0000 UTC m=+1319.051108220"
Nov 02 21:30:31.811544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:31.811504    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 02 21:30:31.811691 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:31.811553    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 02 21:30:32.639610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.639535    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"]
Nov 02 21:30:32.643640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.643617    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.681036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.681009    4528 generic.go:356] "Generic (PLEG): container finished" podID="a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24" containerID="93104cb5c471788a2328e1c85dac92fd665940709c676524a346a0a1b7af9a94" exitCode=0
Nov 02 21:30:32.681216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.681073    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz" event={"ID":"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24","Type":"ContainerDied","Data":"93104cb5c471788a2328e1c85dac92fd665940709c676524a346a0a1b7af9a94"}
Nov 02 21:30:32.692853 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.692821    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-190dba28-mdgs2\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.693064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.693042    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk92v\" (UniqueName: \"kubernetes.io/projected/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-kube-api-access-kk92v\") pod \"maas-api-revoke-sub-190dba28-mdgs2\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.693217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.693185    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-190dba28-mdgs2\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.698500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.698475    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"]
Nov 02 21:30:32.776768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.776739    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:32.777100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.777079    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:32.777659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.777642    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:30:32.794389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.794359    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-190dba28-mdgs2\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.794504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.794493    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk92v\" (UniqueName: \"kubernetes.io/projected/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-kube-api-access-kk92v\") pod \"maas-api-revoke-sub-190dba28-mdgs2\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.794649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.794625    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-190dba28-mdgs2\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.795258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.795236    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-190dba28-mdgs2\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.796652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.796634    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-190dba28-mdgs2\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.814409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.814385    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:32.814665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.814648    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:32.815239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.815224    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:30:32.851892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.851873    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk92v\" (UniqueName: \"kubernetes.io/projected/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-kube-api-access-kk92v\") pod \"maas-api-revoke-sub-190dba28-mdgs2\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:32.956684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:32.956607    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:33.050214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:33.050165    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"]
Nov 02 21:30:33.053490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:33.053466    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4a4142-plrl4"]
Nov 02 21:30:33.153988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:33.153530    4528 scope.go:122] "RemoveContainer" containerID="9aca9a93cab1c2c0537badca3f80b66b814add25cc181802185c1ff3b1acc3a0"
Nov 02 21:30:33.166236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:33.166215    4528 scope.go:122] "RemoveContainer" containerID="1711e1288218a159bee4ae6ad1c255443f79cf2e3cddd1bc6a149141cbb0e03e"
Nov 02 21:30:33.345019 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:33.344986    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6025d5_7b33_44ed_93d4_4ce861ae88e8.slice/crio-9a3d8dcdcf89e13b398c2794e2f0ab8ac068adcf309b0d35e31a3e414e1624d1 WatchSource:0}: Error finding container 9a3d8dcdcf89e13b398c2794e2f0ab8ac068adcf309b0d35e31a3e414e1624d1: Status 404 returned error can't find the container with id 9a3d8dcdcf89e13b398c2794e2f0ab8ac068adcf309b0d35e31a3e414e1624d1
Nov 02 21:30:33.349964 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:33.349938    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"]
Nov 02 21:30:33.690311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:33.690266    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2" event={"ID":"2a6025d5-7b33-44ed-93d4-4ce861ae88e8","Type":"ContainerStarted","Data":"b4d46866061ac7a52784a7b4d978481b74345714e449f98f8a16ba12b7a61c04"}
Nov 02 21:30:33.690767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:33.690364    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2" event={"ID":"2a6025d5-7b33-44ed-93d4-4ce861ae88e8","Type":"ContainerStarted","Data":"9a3d8dcdcf89e13b398c2794e2f0ab8ac068adcf309b0d35e31a3e414e1624d1"}
Nov 02 21:30:33.827019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:33.826960    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2" podStartSLOduration=1.826916784 podStartE2EDuration="1.826916784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:33.818991126 +0000 UTC m=+1321.082527445" watchObservedRunningTime="2026-11-02 21:30:33.826916784 +0000 UTC m=+1321.090453101"
Nov 02 21:30:34.033566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.033541    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:34.104065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.103985    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-kube-api-access-2w4gq\" (UniqueName: \"kubernetes.io/projected/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-kube-api-access-2w4gq\") pod \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") "
Nov 02 21:30:34.104244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.104126    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-openshift-ca\") pod \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") "
Nov 02 21:30:34.104244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.104168    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-xks-ca\") pod \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\" (UID: \"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24\") "
Nov 02 21:30:34.104708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.104676    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-openshift-ca" pod "a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24" (UID: "a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:30:34.106250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.106222    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-kube-api-access-2w4gq" pod "a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24" (UID: "a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24"). InnerVolumeSpecName "kube-api-access-2w4gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:30:34.106377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.106357    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-xks-ca" pod "a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24" (UID: "a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:30:34.206095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.206058    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:34.206095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.206097    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:34.206330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.206142    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w4gq\" (UniqueName: \"kubernetes.io/projected/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24-kube-api-access-2w4gq\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:34.699235 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.699196    4528 generic.go:356] "Generic (PLEG): container finished" podID="2a6025d5-7b33-44ed-93d4-4ce861ae88e8" containerID="b4d46866061ac7a52784a7b4d978481b74345714e449f98f8a16ba12b7a61c04" exitCode=0
Nov 02 21:30:34.699656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.699276    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2" event={"ID":"2a6025d5-7b33-44ed-93d4-4ce861ae88e8","Type":"ContainerDied","Data":"b4d46866061ac7a52784a7b4d978481b74345714e449f98f8a16ba12b7a61c04"}
Nov 02 21:30:34.700779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.700751    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz" event={"ID":"a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24","Type":"ContainerDied","Data":"6a01c6f4f106924940c74b24ccf8a4516deecf20b5dbbc48b045b2e0118fa0a1"}
Nov 02 21:30:34.700779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.700777    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"
Nov 02 21:30:34.700950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.700802    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a01c6f4f106924940c74b24ccf8a4516deecf20b5dbbc48b045b2e0118fa0a1"
Nov 02 21:30:34.915200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:34.915169    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9309246b-b945-4d1a-825d-c206032bca75" path="/var/lib/kubelet/pods/9309246b-b945-4d1a-825d-c206032bca75/volumes"
Nov 02 21:30:35.870415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:35.870392    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:35.932105 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:35.932070    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-xks-ca\") pod \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") "
Nov 02 21:30:35.932315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:35.932237    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-kube-api-access-kk92v\" (UniqueName: \"kubernetes.io/projected/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-kube-api-access-kk92v\") pod \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") "
Nov 02 21:30:35.932315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:35.932291    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-openshift-ca\") pod \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\" (UID: \"2a6025d5-7b33-44ed-93d4-4ce861ae88e8\") "
Nov 02 21:30:35.932704 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:35.932675    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-openshift-ca" pod "2a6025d5-7b33-44ed-93d4-4ce861ae88e8" (UID: "2a6025d5-7b33-44ed-93d4-4ce861ae88e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:30:35.934241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:35.934220    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-xks-ca" pod "2a6025d5-7b33-44ed-93d4-4ce861ae88e8" (UID: "2a6025d5-7b33-44ed-93d4-4ce861ae88e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:30:35.934392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:35.934376    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-kube-api-access-kk92v" pod "2a6025d5-7b33-44ed-93d4-4ce861ae88e8" (UID: "2a6025d5-7b33-44ed-93d4-4ce861ae88e8"). InnerVolumeSpecName "kube-api-access-kk92v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:30:36.034119 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:36.034086    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:36.034284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:36.034142    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk92v\" (UniqueName: \"kubernetes.io/projected/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-kube-api-access-kk92v\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:36.034284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:36.034155    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6025d5-7b33-44ed-93d4-4ce861ae88e8-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:36.736832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:36.736777    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2" event={"ID":"2a6025d5-7b33-44ed-93d4-4ce861ae88e8","Type":"ContainerDied","Data":"9a3d8dcdcf89e13b398c2794e2f0ab8ac068adcf309b0d35e31a3e414e1624d1"}
Nov 02 21:30:36.737013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:36.736870    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a3d8dcdcf89e13b398c2794e2f0ab8ac068adcf309b0d35e31a3e414e1624d1"
Nov 02 21:30:36.737013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:36.736940    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"
Nov 02 21:30:41.291849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.291812    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"]
Nov 02 21:30:41.292425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.292410    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24" containerName="revoke-keys"
Nov 02 21:30:41.292470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.292427    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24" containerName="revoke-keys"
Nov 02 21:30:41.292470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.292438    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2a6025d5-7b33-44ed-93d4-4ce861ae88e8" containerName="revoke-keys"
Nov 02 21:30:41.292470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.292443    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6025d5-7b33-44ed-93d4-4ce861ae88e8" containerName="revoke-keys"
Nov 02 21:30:41.292560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.292533    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24" containerName="revoke-keys"
Nov 02 21:30:41.292560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.292540    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6025d5-7b33-44ed-93d4-4ce861ae88e8" containerName="revoke-keys"
Nov 02 21:30:41.297271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.297255    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.330165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.330144    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0396ef3d-gbrhz\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.330311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.330291    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56hhq\" (UniqueName: \"kubernetes.io/projected/9de84cbf-26d8-48a2-a432-260791fc5c74-kube-api-access-56hhq\") pod \"maas-api-revoke-sub-0396ef3d-gbrhz\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.330451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.330438    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0396ef3d-gbrhz\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.351803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.351775    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"]
Nov 02 21:30:41.395416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.395389    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:30:41.395548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.395440    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:30:41.395548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.395440    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:30:41.401242 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.401222    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:30:41.431878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.431854    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0396ef3d-gbrhz\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.431986 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.431901    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0396ef3d-gbrhz\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.432034 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.432019    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56hhq\" (UniqueName: \"kubernetes.io/projected/9de84cbf-26d8-48a2-a432-260791fc5c74-kube-api-access-56hhq\") pod \"maas-api-revoke-sub-0396ef3d-gbrhz\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.432554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.432535    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0396ef3d-gbrhz\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.434405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.434386    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0396ef3d-gbrhz\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.488592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.488565    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56hhq\" (UniqueName: \"kubernetes.io/projected/9de84cbf-26d8-48a2-a432-260791fc5c74-kube-api-access-56hhq\") pod \"maas-api-revoke-sub-0396ef3d-gbrhz\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.610568 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.610486    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:41.981047 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:30:41.981011    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de84cbf_26d8_48a2_a432_260791fc5c74.slice/crio-866dfc22dd4ffd960c933d0939fa0479f7a04b2ab2a488440b0b8e532ec75d88 WatchSource:0}: Error finding container 866dfc22dd4ffd960c933d0939fa0479f7a04b2ab2a488440b0b8e532ec75d88: Status 404 returned error can't find the container with id 866dfc22dd4ffd960c933d0939fa0479f7a04b2ab2a488440b0b8e532ec75d88
Nov 02 21:30:41.984869 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:41.984841    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"]
Nov 02 21:30:42.777750 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:42.777712    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz" event={"ID":"9de84cbf-26d8-48a2-a432-260791fc5c74","Type":"ContainerStarted","Data":"7342e82de7777cee60150b8c925a2b98fd04b77562419dc608d8990d991a474c"}
Nov 02 21:30:42.778264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:42.777765    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz" event={"ID":"9de84cbf-26d8-48a2-a432-260791fc5c74","Type":"ContainerStarted","Data":"866dfc22dd4ffd960c933d0939fa0479f7a04b2ab2a488440b0b8e532ec75d88"}
Nov 02 21:30:42.895606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:42.895550    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz" podStartSLOduration=1.8955367 podStartE2EDuration="1.8955367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:42.891433557 +0000 UTC m=+1330.154969873" watchObservedRunningTime="2026-11-02 21:30:42.8955367 +0000 UTC m=+1330.159073017"
Nov 02 21:30:43.785586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:43.785552    4528 generic.go:356] "Generic (PLEG): container finished" podID="9de84cbf-26d8-48a2-a432-260791fc5c74" containerID="7342e82de7777cee60150b8c925a2b98fd04b77562419dc608d8990d991a474c" exitCode=0
Nov 02 21:30:43.786020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:43.785594    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz" event={"ID":"9de84cbf-26d8-48a2-a432-260791fc5c74","Type":"ContainerDied","Data":"7342e82de7777cee60150b8c925a2b98fd04b77562419dc608d8990d991a474c"}
Nov 02 21:30:44.113144 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:30:44.113048    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:30:45.243906 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.243876    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:45.336370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.336334    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-openshift-ca\") pod \"9de84cbf-26d8-48a2-a432-260791fc5c74\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") "
Nov 02 21:30:45.336645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.336626    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-xks-ca\") pod \"9de84cbf-26d8-48a2-a432-260791fc5c74\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") "
Nov 02 21:30:45.336779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.336765    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de84cbf-26d8-48a2-a432-260791fc5c74-kube-api-access-56hhq\" (UniqueName: \"kubernetes.io/projected/9de84cbf-26d8-48a2-a432-260791fc5c74-kube-api-access-56hhq\") pod \"9de84cbf-26d8-48a2-a432-260791fc5c74\" (UID: \"9de84cbf-26d8-48a2-a432-260791fc5c74\") "
Nov 02 21:30:45.338329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.338302    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-openshift-ca" pod "9de84cbf-26d8-48a2-a432-260791fc5c74" (UID: "9de84cbf-26d8-48a2-a432-260791fc5c74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:30:45.366335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.355317    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de84cbf-26d8-48a2-a432-260791fc5c74-kube-api-access-56hhq" pod "9de84cbf-26d8-48a2-a432-260791fc5c74" (UID: "9de84cbf-26d8-48a2-a432-260791fc5c74"). InnerVolumeSpecName "kube-api-access-56hhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:30:45.366335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.355856    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-xks-ca" pod "9de84cbf-26d8-48a2-a432-260791fc5c74" (UID: "9de84cbf-26d8-48a2-a432-260791fc5c74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:30:45.445185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.439325    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:45.445185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.439365    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de84cbf-26d8-48a2-a432-260791fc5c74-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:45.445185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.439381    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56hhq\" (UniqueName: \"kubernetes.io/projected/9de84cbf-26d8-48a2-a432-260791fc5c74-kube-api-access-56hhq\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:30:45.854648 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.854596    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz" event={"ID":"9de84cbf-26d8-48a2-a432-260791fc5c74","Type":"ContainerDied","Data":"866dfc22dd4ffd960c933d0939fa0479f7a04b2ab2a488440b0b8e532ec75d88"}
Nov 02 21:30:45.854897 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.854878    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866dfc22dd4ffd960c933d0939fa0479f7a04b2ab2a488440b0b8e532ec75d88"
Nov 02 21:30:45.855101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:45.855087    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"
Nov 02 21:30:46.753250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:46.753204    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:46.839135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:46.829263    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:47.145130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:47.144357    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:47.205133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:47.204953    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:47.376485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:47.376432    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:47.449363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:47.449252    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:47.761192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:47.760202    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 02 21:30:47.761192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:47.760842    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:30:47.873624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:47.849263    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:47.873624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:47.851185    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:30:48.263549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:48.263519    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:30:48.371577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:48.363799    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:30:48.429618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:48.424526    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:48.429618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:48.425242    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:30:48.434834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:48.434528    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Nov 02 21:30:48.868722 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:48.868264    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:30:48.869379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:48.869332    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:49.309375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:49.309321    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:49.356469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:49.353555    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded"
Nov 02 21:30:49.605184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:49.586709    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 02 21:30:49.605184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:49.588528    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 02 21:30:49.633131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:49.631178    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:30:49.813431 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:30:49.813397    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:31:02.571916 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.571867    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"]
Nov 02 21:31:02.573521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.573482    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9de84cbf-26d8-48a2-a432-260791fc5c74" containerName="revoke-keys"
Nov 02 21:31:02.573692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.573674    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de84cbf-26d8-48a2-a432-260791fc5c74" containerName="revoke-keys"
Nov 02 21:31:02.573993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.573951    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de84cbf-26d8-48a2-a432-260791fc5c74" containerName="revoke-keys"
Nov 02 21:31:02.587720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.587681    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:02.614899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.614804    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e06a0-bc8e-4cb5-b81c-b85942339178-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj\" (UID: \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:02.615060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.614947    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84d5c\" (UniqueName: \"kubernetes.io/projected/9b0e06a0-bc8e-4cb5-b81c-b85942339178-kube-api-access-84d5c\") pod \"payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj\" (UID: \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:02.623947 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.623918    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"]
Nov 02 21:31:02.671145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.670514    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d58d18\""
Nov 02 21:31:02.671145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.670715    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d58d18-dockercfg-txjv8\""
Nov 02 21:31:02.776133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.775367    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e06a0-bc8e-4cb5-b81c-b85942339178-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj\" (UID: \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:02.776133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.775536    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d5c\" (UniqueName: \"kubernetes.io/projected/9b0e06a0-bc8e-4cb5-b81c-b85942339178-kube-api-access-84d5c\") pod \"payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj\" (UID: \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:02.789131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.777219    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e06a0-bc8e-4cb5-b81c-b85942339178-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj\" (UID: \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:02.850168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.849290    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84d5c\" (UniqueName: \"kubernetes.io/projected/9b0e06a0-bc8e-4cb5-b81c-b85942339178-kube-api-access-84d5c\") pod \"payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj\" (UID: \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:02.910289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:02.910258    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:03.289387 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:31:03.289354    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0e06a0_bc8e_4cb5_b81c_b85942339178.slice/crio-b88a59d34a937f4b08b43438d3d314b475507ac427df1ec350735d226ea2b118 WatchSource:0}: Error finding container b88a59d34a937f4b08b43438d3d314b475507ac427df1ec350735d226ea2b118: Status 404 returned error can't find the container with id b88a59d34a937f4b08b43438d3d314b475507ac427df1ec350735d226ea2b118
Nov 02 21:31:03.293189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:03.293167    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"]
Nov 02 21:31:03.844148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:03.844092    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj" event={"ID":"9b0e06a0-bc8e-4cb5-b81c-b85942339178","Type":"ContainerStarted","Data":"e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926"}
Nov 02 21:31:03.844602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:03.844167    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj" event={"ID":"9b0e06a0-bc8e-4cb5-b81c-b85942339178","Type":"ContainerStarted","Data":"b88a59d34a937f4b08b43438d3d314b475507ac427df1ec350735d226ea2b118"}
Nov 02 21:31:03.844602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:03.844323    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:03.962728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:03.962675    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj" podStartSLOduration=1.962660244 podStartE2EDuration="1.962660244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:03.960403334 +0000 UTC m=+1351.223939653" watchObservedRunningTime="2026-11-02 21:31:03.962660244 +0000 UTC m=+1351.226196561"
Nov 02 21:31:04.565515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.565484    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"]
Nov 02 21:31:04.569670 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.569651    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.619133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.613607    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-credential-socket\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.619133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.613690    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-token\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.619133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.613739    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-data\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.619133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.613766    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istiod-ca-cert\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.619133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.613824    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vknx4\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-kube-api-access-vknx4\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.619133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.613856    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-podinfo\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.619133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.613910    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-envoy\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.619133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.613985    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-socket\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.619133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.614055    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-certs\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.760134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.690538    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"]
Nov 02 21:31:04.760134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.694520    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-03ece2-openshift-default-dockercfg-lfvvt\""
Nov 02 21:31:04.803141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.795605    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-token\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.803141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.795694    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-data\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.803141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.795735    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istiod-ca-cert\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.803141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.795828    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vknx4\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-kube-api-access-vknx4\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.803141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.799031    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-data\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.803141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.799974    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istiod-ca-cert\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.807421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.806322    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-podinfo\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.816765 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.816288    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-podinfo\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.816765 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.816458    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-envoy\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.829818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.829509    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-socket\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.829818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.829616    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-certs\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.829818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.829673    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-credential-socket\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.830127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.830063    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-credential-socket\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.833352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.830360    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-socket\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.833352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.830582    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-certs\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.876131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.874070    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vknx4\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-kube-api-access-vknx4\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.885886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.885853    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-token\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:04.889273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:04.889217    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-envoy\") pod \"e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") " pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:05.184466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:05.184335    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:05.623543 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:31:05.623509    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ed76ea_1ce8_4b37_b55c_5c98e5eef187.slice/crio-0759927c54a177be1d8a6059bb394010de0d8b9221dfc76f8348c0cb38032338 WatchSource:0}: Error finding container 0759927c54a177be1d8a6059bb394010de0d8b9221dfc76f8348c0cb38032338: Status 404 returned error can't find the container with id 0759927c54a177be1d8a6059bb394010de0d8b9221dfc76f8348c0cb38032338
Nov 02 21:31:05.626096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:05.626064    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:05.626205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:05.626150    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:05.626205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:05.626196    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:05.627400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:05.627378    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"]
Nov 02 21:31:05.917239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:05.916102    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" event={"ID":"36ed76ea-1ce8-4b37-b55c-5c98e5eef187","Type":"ContainerStarted","Data":"2c598e39c01a75d199878f0c10aeb61eda88c8ffba6d1aa8c39cf3ab13e4cae4"}
Nov 02 21:31:05.917239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:05.916187    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" event={"ID":"36ed76ea-1ce8-4b37-b55c-5c98e5eef187","Type":"ContainerStarted","Data":"0759927c54a177be1d8a6059bb394010de0d8b9221dfc76f8348c0cb38032338"}
Nov 02 21:31:06.036902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:06.036842    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" podStartSLOduration=2.036828376 podStartE2EDuration="2.036828376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:06.033208815 +0000 UTC m=+1353.296745135" watchObservedRunningTime="2026-11-02 21:31:06.036828376 +0000 UTC m=+1353.300364692"
Nov 02 21:31:06.189824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:06.189720    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:06.227133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:06.226878    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:06.922812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:06.922780    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:06.923795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:06.923779    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:31:07.893369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:07.893329    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:31:14.113325 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:31:14.113280    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:31:14.884505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:14.884476    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:31:25.406544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.406459    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"]
Nov 02 21:31:25.410781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.410760    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.517672 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.517630    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"]
Nov 02 21:31:25.546501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.546467    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s9rk\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-kube-api-access-6s9rk\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.546668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.546529    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.546668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.546631    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.546740 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.546667    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.546773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.546755    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0355ee02-9c37-473d-9555-e60358738c22-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.546808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.546793    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.546841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.546833    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.546955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.546934    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.546990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.546980    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0355ee02-9c37-473d-9555-e60358738c22-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.647994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.647959    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648049    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648078    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0355ee02-9c37-473d-9555-e60358738c22-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648182    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s9rk\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-kube-api-access-6s9rk\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648246    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648295    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648326    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648379    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0355ee02-9c37-473d-9555-e60358738c22-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648384    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648427    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648482    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648848    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.648870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.648852    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.649052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.649007    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0355ee02-9c37-473d-9555-e60358738c22-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.650680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.650661    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.650813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.650797    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0355ee02-9c37-473d-9555-e60358738c22-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.705301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.705236    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.705407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.705390    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s9rk\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-kube-api-access-6s9rk\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:25.724220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:25.724198    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:26.097762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.097733    4528 generic.go:356] "Generic (PLEG): container finished" podID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerID="ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150" exitCode=0
Nov 02 21:31:26.097928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.097802    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" event={"ID":"2ae9bbaf-405b-43a8-b353-73a2265a1ad3","Type":"ContainerDied","Data":"ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150"}
Nov 02 21:31:26.098216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.098202    4528 scope.go:122] "RemoveContainer" containerID="ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150"
Nov 02 21:31:26.099543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.099518    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:26.099605 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.099583    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:26.099643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.099615    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:26.145626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.145604    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"]
Nov 02 21:31:26.176664 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:31:26.176631    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0355ee02_9c37_473d_9555_e60358738c22.slice/crio-38cba048a90bcd60f02bff82106d5a1ad2356dff199bca37f42262337b38322c WatchSource:0}: Error finding container 38cba048a90bcd60f02bff82106d5a1ad2356dff199bca37f42262337b38322c: Status 404 returned error can't find the container with id 38cba048a90bcd60f02bff82106d5a1ad2356dff199bca37f42262337b38322c
Nov 02 21:31:26.179754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.179722    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:26.179882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.179805    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:26.179882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:26.179846    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:27.154149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.153100    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" event={"ID":"2ae9bbaf-405b-43a8-b353-73a2265a1ad3","Type":"ContainerStarted","Data":"7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca"}
Nov 02 21:31:27.155153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.155094    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" event={"ID":"0355ee02-9c37-473d-9555-e60358738c22","Type":"ContainerStarted","Data":"b2c20a520ca5137b499077c762d67f69390391ca1bc5c49875d3307fdf96d591"}
Nov 02 21:31:27.155258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.155172    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" event={"ID":"0355ee02-9c37-473d-9555-e60358738c22","Type":"ContainerStarted","Data":"38cba048a90bcd60f02bff82106d5a1ad2356dff199bca37f42262337b38322c"}
Nov 02 21:31:27.340131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.339692    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:31:27.341249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.341222    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 21:31:27.341463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.341442    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 21:31:27.416134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.403752    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" podStartSLOduration=2.403733381 podStartE2EDuration="2.403733381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:27.40251129 +0000 UTC m=+1374.666047607" watchObservedRunningTime="2026-11-02 21:31:27.403733381 +0000 UTC m=+1374.667269698"
Nov 02 21:31:27.729240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.728166    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:27.731268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.731233    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 02 21:31:27.731509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:27.731479    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 02 21:31:28.332134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:28.331552    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 21:31:28.332134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:28.331617    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 21:31:28.729189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:28.728482    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 02 21:31:28.729189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:28.728543    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 02 21:31:29.334134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:29.333099    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 21:31:29.334134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:29.333174    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 21:31:29.727189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:29.726503    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 02 21:31:29.727189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:29.726567    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 02 21:31:30.335982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:30.335532    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 21:31:30.335982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:30.335595    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 21:31:30.562628 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:30.562594    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:31:30.729191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:30.728556    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 02 21:31:30.729191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:30.728629    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 02 21:31:31.340580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:31.340539    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 21:31:31.341251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:31.341222    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 21:31:31.727186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:31.726565    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 02 21:31:31.727186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:31.726642    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 02 21:31:32.335379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:32.335349    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:31:32.336678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:32.336653    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:31:32.725818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:32.725722    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 02 21:31:32.725818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:32.725796    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 02 21:31:33.222241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:33.222202    4528 scope.go:122] "RemoveContainer" containerID="c27f32c7a7803338690e057fd5f5da891d702cff8f9a1c4dc17d59ff57efe6bd"
Nov 02 21:31:33.231958 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:33.231926    4528 scope.go:122] "RemoveContainer" containerID="dc54ac4779fd76ee780253a8c88d9a006dce8d0aa52d091f49102cdad63ba105"
Nov 02 21:31:33.243386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:33.243291    4528 scope.go:122] "RemoveContainer" containerID="539acbb82e10e006695530023eda6795974116193f9bd7995fbc73e46d1923f2"
Nov 02 21:31:33.261955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:33.261933    4528 scope.go:122] "RemoveContainer" containerID="cbfd66d311c2bdffb24763f02165e094ecb51b6db64656d480f99a553c5bdedf"
Nov 02 21:31:33.273875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:33.273851    4528 scope.go:122] "RemoveContainer" containerID="e9d3cbe856cb98feeb9749e0af1fd6586e6e409ac8c276bca107ac2685c57656"
Nov 02 21:31:33.285149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:33.285131    4528 scope.go:122] "RemoveContainer" containerID="4cc8a09a312ee5192188bc8c468a39e97821ef643b718e0b9e4eefbee51267af"
Nov 02 21:31:33.728794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:33.728768    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:33.729374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:33.729286    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:33.729901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:33.729881    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:31:38.637045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.637008    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"]
Nov 02 21:31:38.643201 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.643184    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.659666 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.659640    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531e47f-f6df-48cd-948d-78b67df65b79-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.659799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.659687    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.659799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.659710    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.659799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.659765    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531e47f-f6df-48cd-948d-78b67df65b79-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.659916 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.659886    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.659966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.659954    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.660015 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.660004    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.660089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.660077    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.660174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.660125    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzglz\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-kube-api-access-rzglz\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.698318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.698284    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f51151-openshift-default-dockercfg-5nn46\""
Nov 02 21:31:38.752723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.752685    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"]
Nov 02 21:31:38.761013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.760976    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.761189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.761023    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.761189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.761047    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531e47f-f6df-48cd-948d-78b67df65b79-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.761189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.761085    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.761189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.761178    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.761425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.761253    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.761425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.761347    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.761425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.761374    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzglz\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-kube-api-access-rzglz\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.761577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.761463    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531e47f-f6df-48cd-948d-78b67df65b79-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.761763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.761735    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.762059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.762009    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.762190 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.762078    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.762296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.762276    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.763653 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.763580    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531e47f-f6df-48cd-948d-78b67df65b79-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.764269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.764246    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.764356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.764290    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531e47f-f6df-48cd-948d-78b67df65b79-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.819365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.819339    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.819503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.819406    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzglz\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-kube-api-access-rzglz\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:38.957091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:38.957009    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:39.378349 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:31:39.378310    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4531e47f_f6df_48cd_948d_78b67df65b79.slice/crio-8049714bf81f83e822532fd88f6bf3ab88637adaaf01abe809279bf28f66d777 WatchSource:0}: Error finding container 8049714bf81f83e822532fd88f6bf3ab88637adaaf01abe809279bf28f66d777: Status 404 returned error can't find the container with id 8049714bf81f83e822532fd88f6bf3ab88637adaaf01abe809279bf28f66d777
Nov 02 21:31:39.380849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:39.380813    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:39.380934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:39.380897    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:39.380992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:39.380963    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:31:39.383392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:39.383361    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"]
Nov 02 21:31:40.246780 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:40.246746    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" event={"ID":"4531e47f-f6df-48cd-948d-78b67df65b79","Type":"ContainerStarted","Data":"2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d"}
Nov 02 21:31:40.247282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:40.246803    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" event={"ID":"4531e47f-f6df-48cd-948d-78b67df65b79","Type":"ContainerStarted","Data":"8049714bf81f83e822532fd88f6bf3ab88637adaaf01abe809279bf28f66d777"}
Nov 02 21:31:40.509905 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:40.494887    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" podStartSLOduration=2.49486857 podStartE2EDuration="2.49486857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:40.423615965 +0000 UTC m=+1387.687152282" watchObservedRunningTime="2026-11-02 21:31:40.49486857 +0000 UTC m=+1387.758404888"
Nov 02 21:31:40.957451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:40.957417    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:40.959238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:40.959212    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 21:31:40.959364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:40.959258    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 21:31:41.957860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:41.957818    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 21:31:41.958393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:41.957876    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 21:31:42.957735 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:42.957697    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 21:31:42.957940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:42.957751    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 21:31:43.957793 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:43.957757    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 21:31:43.958031 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:43.957813    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 21:31:44.134698 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:31:44.134658    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:31:44.957897 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:44.957824    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 21:31:44.958052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:44.957912    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 21:31:45.957338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:45.957296    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 21:31:45.957533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:45.957368    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 21:31:46.962187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:46.962147    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:46.962694 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:46.962489    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:46.963137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:46.963089    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:31:53.202214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.202180    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"]
Nov 02 21:31:53.212363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.212329    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.259710 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.259681    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"]
Nov 02 21:31:53.305065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.305027    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:31:53.305065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.305049    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:31:53.315699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.315676    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:31:53.315699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.315677    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:31:53.336734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.336710    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-380a24a3-6bvbg\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.336854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.336837    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8zw\" (UniqueName: \"kubernetes.io/projected/d7a82826-97c5-4ba6-b32f-9d78606744a3-kube-api-access-fk8zw\") pod \"maas-api-revoke-sub-380a24a3-6bvbg\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.336962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.336930    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-380a24a3-6bvbg\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.438154 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.438089    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-380a24a3-6bvbg\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.438354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.438236    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-380a24a3-6bvbg\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.438354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.438340    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8zw\" (UniqueName: \"kubernetes.io/projected/d7a82826-97c5-4ba6-b32f-9d78606744a3-kube-api-access-fk8zw\") pod \"maas-api-revoke-sub-380a24a3-6bvbg\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.438926 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.438898    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-380a24a3-6bvbg\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.440565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.440540    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-380a24a3-6bvbg\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.495180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.495087    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8zw\" (UniqueName: \"kubernetes.io/projected/d7a82826-97c5-4ba6-b32f-9d78606744a3-kube-api-access-fk8zw\") pod \"maas-api-revoke-sub-380a24a3-6bvbg\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.527600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.527568    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:53.896565 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:31:53.896530    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a82826_97c5_4ba6_b32f_9d78606744a3.slice/crio-335c113b63fd3e96a7d0e91f32e592f096352d112537af9c13fae9b0b23d81d2 WatchSource:0}: Error finding container 335c113b63fd3e96a7d0e91f32e592f096352d112537af9c13fae9b0b23d81d2: Status 404 returned error can't find the container with id 335c113b63fd3e96a7d0e91f32e592f096352d112537af9c13fae9b0b23d81d2
Nov 02 21:31:53.900787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:53.900764    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"]
Nov 02 21:31:54.436102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:54.436066    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg" event={"ID":"d7a82826-97c5-4ba6-b32f-9d78606744a3","Type":"ContainerStarted","Data":"90700e2b8079bdc3c1cf5e48ded922f0c2e3edbe1ec482a6e9dcdfcaf0b5cf05"}
Nov 02 21:31:54.436553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:54.436146    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg" event={"ID":"d7a82826-97c5-4ba6-b32f-9d78606744a3","Type":"ContainerStarted","Data":"335c113b63fd3e96a7d0e91f32e592f096352d112537af9c13fae9b0b23d81d2"}
Nov 02 21:31:54.553527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:54.553475    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg" podStartSLOduration=1.553460617 podStartE2EDuration="1.553460617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:54.549747662 +0000 UTC m=+1401.813283981" watchObservedRunningTime="2026-11-02 21:31:54.553460617 +0000 UTC m=+1401.816996934"
Nov 02 21:31:55.444408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:55.444371    4528 generic.go:356] "Generic (PLEG): container finished" podID="d7a82826-97c5-4ba6-b32f-9d78606744a3" containerID="90700e2b8079bdc3c1cf5e48ded922f0c2e3edbe1ec482a6e9dcdfcaf0b5cf05" exitCode=0
Nov 02 21:31:55.444781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:55.444416    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg" event={"ID":"d7a82826-97c5-4ba6-b32f-9d78606744a3","Type":"ContainerDied","Data":"90700e2b8079bdc3c1cf5e48ded922f0c2e3edbe1ec482a6e9dcdfcaf0b5cf05"}
Nov 02 21:31:56.767034 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:56.767004    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:56.891879 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:56.891797    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-openshift-ca\") pod \"d7a82826-97c5-4ba6-b32f-9d78606744a3\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") "
Nov 02 21:31:56.892093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:56.892073    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-xks-ca\") pod \"d7a82826-97c5-4ba6-b32f-9d78606744a3\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") "
Nov 02 21:31:56.892185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:56.892165    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-openshift-ca" pod "d7a82826-97c5-4ba6-b32f-9d78606744a3" (UID: "d7a82826-97c5-4ba6-b32f-9d78606744a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:31:56.892240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:56.892210    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a82826-97c5-4ba6-b32f-9d78606744a3-kube-api-access-fk8zw\" (UniqueName: \"kubernetes.io/projected/d7a82826-97c5-4ba6-b32f-9d78606744a3-kube-api-access-fk8zw\") pod \"d7a82826-97c5-4ba6-b32f-9d78606744a3\" (UID: \"d7a82826-97c5-4ba6-b32f-9d78606744a3\") "
Nov 02 21:31:56.892744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:56.892723    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:31:56.894258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:56.894230    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-xks-ca" pod "d7a82826-97c5-4ba6-b32f-9d78606744a3" (UID: "d7a82826-97c5-4ba6-b32f-9d78606744a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:31:56.894695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:56.894679    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a82826-97c5-4ba6-b32f-9d78606744a3-kube-api-access-fk8zw" pod "d7a82826-97c5-4ba6-b32f-9d78606744a3" (UID: "d7a82826-97c5-4ba6-b32f-9d78606744a3"). InnerVolumeSpecName "kube-api-access-fk8zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:31:57.002031 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:57.001921    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7a82826-97c5-4ba6-b32f-9d78606744a3-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:31:57.002291 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:57.002263    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8zw\" (UniqueName: \"kubernetes.io/projected/d7a82826-97c5-4ba6-b32f-9d78606744a3-kube-api-access-fk8zw\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:31:57.461529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:57.461494    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg" event={"ID":"d7a82826-97c5-4ba6-b32f-9d78606744a3","Type":"ContainerDied","Data":"335c113b63fd3e96a7d0e91f32e592f096352d112537af9c13fae9b0b23d81d2"}
Nov 02 21:31:57.461705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:57.461548    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"
Nov 02 21:31:57.461705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:31:57.461565    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="335c113b63fd3e96a7d0e91f32e592f096352d112537af9c13fae9b0b23d81d2"
Nov 02 21:32:01.147911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:01.147867    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:32:10.323231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.323199    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"]
Nov 02 21:32:10.323805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.323789    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d7a82826-97c5-4ba6-b32f-9d78606744a3" containerName="revoke-keys"
Nov 02 21:32:10.323849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.323806    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a82826-97c5-4ba6-b32f-9d78606744a3" containerName="revoke-keys"
Nov 02 21:32:10.323897 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.323889    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a82826-97c5-4ba6-b32f-9d78606744a3" containerName="revoke-keys"
Nov 02 21:32:10.327714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.327692    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.377496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.377461    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"]
Nov 02 21:32:10.382484 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.382462    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.385631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.385615    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d58d18-openshift-default-dockercfg-6dd47\""
Nov 02 21:32:10.424184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.424157    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"]
Nov 02 21:32:10.445379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445344    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.445542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445433    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ls9v\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-kube-api-access-9ls9v\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.445542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445461    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdlbs\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-kube-api-access-gdlbs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.445542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445513    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.445715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445595    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52228fc8-539f-40bd-9f70-9c08cfed5739-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.445715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445656    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.445715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445679    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.445876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445716    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.445876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445788    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.445876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445833    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.445876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445862    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.446082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445901    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.446082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445953    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.446082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.445975    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.446082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.446058    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.446082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.446080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.446414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.446128    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.446414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.446154    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.477544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.477516    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"]
Nov 02 21:32:10.547055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547014    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.547273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547077    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.547273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547123    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.547273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547160    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.547273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547208    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.547273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547229    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.547550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547302    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.547550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547327    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.547550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547357    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.547550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547380    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.547550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547407    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.547550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547475    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ls9v\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-kube-api-access-9ls9v\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.547550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547476    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.547550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547500    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlbs\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-kube-api-access-gdlbs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.547550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547555    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.548003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547642    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52228fc8-539f-40bd-9f70-9c08cfed5739-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.548003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547709    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.548003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547732    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.548003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547770    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.548003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.547928    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.548292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.548099    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.548292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.548202    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.548292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.548236    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.548479 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.548457    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.548849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.548826    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.549598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.549215    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.549598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.549216    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52228fc8-539f-40bd-9f70-9c08cfed5739-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.549932 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.549882    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.550465 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.550437    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.555179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.555148    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.555179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.555148    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.557036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.557014    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.606007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.605938    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.606007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.605977    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ls9v\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-kube-api-access-9ls9v\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.606227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.606041    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.606227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.606211    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlbs\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-kube-api-access-gdlbs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:10.640994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.640972    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:10.699213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:10.698972    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:11.058045 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:32:11.057977    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbc8da9_f9e7_4b71_b25e_71beccf3d298.slice/crio-2fe5bffd680c7ee6a6ee1abcd6a081d007559219a603df2cdd4db8a41a5e286d WatchSource:0}: Error finding container 2fe5bffd680c7ee6a6ee1abcd6a081d007559219a603df2cdd4db8a41a5e286d: Status 404 returned error can't find the container with id 2fe5bffd680c7ee6a6ee1abcd6a081d007559219a603df2cdd4db8a41a5e286d
Nov 02 21:32:11.060464 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.060428    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:11.060580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.060509    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:11.060580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.060553    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:11.062313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.062291    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"]
Nov 02 21:32:11.107243 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:32:11.107213    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52228fc8_539f_40bd_9f70_9c08cfed5739.slice/crio-d7c7234d879e0974d5443520e701f3e2d06a6c49d7f330071622445ad1347702 WatchSource:0}: Error finding container d7c7234d879e0974d5443520e701f3e2d06a6c49d7f330071622445ad1347702: Status 404 returned error can't find the container with id d7c7234d879e0974d5443520e701f3e2d06a6c49d7f330071622445ad1347702
Nov 02 21:32:11.109975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.109944    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:11.110047 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.110023    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:11.110083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.110064    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:11.111497 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.111477    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"]
Nov 02 21:32:11.558401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.558064    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" event={"ID":"6fbc8da9-f9e7-4b71-b25e-71beccf3d298","Type":"ContainerStarted","Data":"fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800"}
Nov 02 21:32:11.558401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.558191    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" event={"ID":"6fbc8da9-f9e7-4b71-b25e-71beccf3d298","Type":"ContainerStarted","Data":"2fe5bffd680c7ee6a6ee1abcd6a081d007559219a603df2cdd4db8a41a5e286d"}
Nov 02 21:32:11.560233 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.560162    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" event={"ID":"52228fc8-539f-40bd-9f70-9c08cfed5739","Type":"ContainerStarted","Data":"8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95"}
Nov 02 21:32:11.560233 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.560212    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" event={"ID":"52228fc8-539f-40bd-9f70-9c08cfed5739","Type":"ContainerStarted","Data":"d7c7234d879e0974d5443520e701f3e2d06a6c49d7f330071622445ad1347702"}
Nov 02 21:32:11.641574 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.641500    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:11.686308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.686249    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" podStartSLOduration=1.686224747 podStartE2EDuration="1.686224747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:11.683205397 +0000 UTC m=+1418.946741713" watchObservedRunningTime="2026-11-02 21:32:11.686224747 +0000 UTC m=+1418.949761062"
Nov 02 21:32:11.713136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.712207    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:11.819204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:11.819061    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" podStartSLOduration=1.8190403769999999 podStartE2EDuration="1.819040377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:11.816487669 +0000 UTC m=+1419.080023987" watchObservedRunningTime="2026-11-02 21:32:11.819040377 +0000 UTC m=+1419.082576696"
Nov 02 21:32:12.647878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:12.647844    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 21:32:12.648474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:12.648440    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 21:32:12.720349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:12.720316    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 21:32:12.720628 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:12.720591    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 21:32:13.642133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:13.641536    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 21:32:13.642133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:13.641616    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 21:32:13.752135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:13.751339    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 21:32:13.752135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:13.751416    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 21:32:14.159130 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:32:14.158807    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:32:14.644130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:14.643495    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 21:32:14.644130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:14.643568    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 21:32:14.714134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:14.705480    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 21:32:14.714134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:14.705541    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 21:32:15.643526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:15.643490    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 21:32:15.644138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:15.644086    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 21:32:15.705472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:15.705437    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 21:32:15.705747 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:15.705720    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 21:32:16.648133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:16.646536    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 21:32:16.648133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:16.646616    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 21:32:16.710131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:16.709458    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 02 21:32:16.710131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:16.709518    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 02 21:32:17.646396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:17.646366    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:17.646802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:17.646786    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:17.647399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:17.647378    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:32:17.709437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:17.709412    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:17.709778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:17.709680    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:17.710234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:17.710215    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:32:27.213191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:27.208969    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:32:27.276133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:27.272187    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:32:27.371133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:27.370430    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:32:32.288351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.288312    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"]
Nov 02 21:32:32.292707 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.292687    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.373616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.373589    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-54a4b2-openshift-default-dockercfg-rnhgj\""
Nov 02 21:32:32.402855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.402816    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"]
Nov 02 21:32:32.416007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.415971    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-socket\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.416219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.416165    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-certs\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.416304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.416260    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-data\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.416366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.416350    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-podinfo\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.416430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.416385    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istiod-ca-cert\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.416560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.416536    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvzj9\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-kube-api-access-mvzj9\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.416636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.416586    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-credential-socket\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.416636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.416611    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-token\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.417058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.416755    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-envoy\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.524774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.524733    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvzj9\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-kube-api-access-mvzj9\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.525132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.525089    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-credential-socket\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.525275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.525259    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-token\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.525436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.525420    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-envoy\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.525627 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.525611    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-socket\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.529225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.525787    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-certs\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.529505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.529479    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-data\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.529756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.529723    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-podinfo\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.529928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.529909    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istiod-ca-cert\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.531190 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.531015    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istiod-ca-cert\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.544262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.543531    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-socket\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.544262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.544165    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-data\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.550258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.545077    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-certs\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.550819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.550791    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-credential-socket\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.575354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.575261    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-podinfo\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.579141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.577865    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-envoy\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.611784 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.611711    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-token\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.626778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.623964    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvzj9\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-kube-api-access-mvzj9\") pod \"e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") " pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:32.958439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.958367    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-54a4b2-openshift-default-dockercfg-rnhgj\""
Nov 02 21:32:32.967665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:32.967644    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:33.397672 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:32:33.397645    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfd76d5_b34a_41e6_badb_43f8a4a9c474.slice/crio-d0bb306842014c4fcaecc9a7dc7759c76610887426c94e6227b806b7b4058406 WatchSource:0}: Error finding container d0bb306842014c4fcaecc9a7dc7759c76610887426c94e6227b806b7b4058406: Status 404 returned error can't find the container with id d0bb306842014c4fcaecc9a7dc7759c76610887426c94e6227b806b7b4058406
Nov 02 21:32:33.400758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:33.400729    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:33.400867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:33.400792    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:33.400867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:33.400820    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:33.401366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:33.401346    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"]
Nov 02 21:32:33.806463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:33.806429    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" event={"ID":"dbfd76d5-b34a-41e6-badb-43f8a4a9c474","Type":"ContainerStarted","Data":"3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7"}
Nov 02 21:32:33.806634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:33.806490    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" event={"ID":"dbfd76d5-b34a-41e6-badb-43f8a4a9c474","Type":"ContainerStarted","Data":"d0bb306842014c4fcaecc9a7dc7759c76610887426c94e6227b806b7b4058406"}
Nov 02 21:32:33.968379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:33.968348    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:33.975632 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:33.975584    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" podStartSLOduration=1.975571022 podStartE2EDuration="1.975571022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:33.973831088 +0000 UTC m=+1441.237367407" watchObservedRunningTime="2026-11-02 21:32:33.975571022 +0000 UTC m=+1441.239107338"
Nov 02 21:32:34.814522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:34.814491    4528 generic.go:356] "Generic (PLEG): container finished" podID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerID="2c598e39c01a75d199878f0c10aeb61eda88c8ffba6d1aa8c39cf3ab13e4cae4" exitCode=0
Nov 02 21:32:34.815018 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:34.814567    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" event={"ID":"36ed76ea-1ce8-4b37-b55c-5c98e5eef187","Type":"ContainerDied","Data":"2c598e39c01a75d199878f0c10aeb61eda88c8ffba6d1aa8c39cf3ab13e4cae4"}
Nov 02 21:32:34.815018 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:34.814979    4528 scope.go:122] "RemoveContainer" containerID="2c598e39c01a75d199878f0c10aeb61eda88c8ffba6d1aa8c39cf3ab13e4cae4"
Nov 02 21:32:34.816296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:34.816273    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:34.816361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:34.816336    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:34.816396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:34.816366    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:34.973790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:34.973759    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:35.831866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:35.831828    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" event={"ID":"36ed76ea-1ce8-4b37-b55c-5c98e5eef187","Type":"ContainerStarted","Data":"3f5fed2b3bb65484fd6c0f0468dbca971f2f9a02758a1765b0439abcdf1e7d07"}
Nov 02 21:32:35.832318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:35.832061    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:35.833246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:35.833225    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:32:36.185932 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:36.185848    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:32:36.186963 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:36.186936    4528 patch_prober.go:29] interesting pod/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 02 21:32:36.187131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:36.186982    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 02 21:32:36.927161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:36.926288    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:32:37.185682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:37.185601    4528 patch_prober.go:29] interesting pod/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 02 21:32:37.185682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:37.185660    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 02 21:32:38.185752 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:38.185717    4528 patch_prober.go:29] interesting pod/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 02 21:32:38.186161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:38.185767    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 02 21:32:39.185893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:39.185859    4528 patch_prober.go:29] interesting pod/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 02 21:32:39.186332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:39.185922    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 02 21:32:40.227137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.212494    4528 patch_prober.go:29] interesting pod/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 02 21:32:40.227137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.212554    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 02 21:32:40.396140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.387289    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"]
Nov 02 21:32:40.409823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.407504    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:40.460136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.444444    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"]
Nov 02 21:32:40.498161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.498035    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-54a4b2-dockercfg-mhxx5\""
Nov 02 21:32:40.536583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.536537    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970f66e4-08da-4bff-8533-1e81a9fddbb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj\" (UID: \"970f66e4-08da-4bff-8533-1e81a9fddbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:40.536931 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.536896    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699fq\" (UniqueName: \"kubernetes.io/projected/970f66e4-08da-4bff-8533-1e81a9fddbb6-kube-api-access-699fq\") pod \"payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj\" (UID: \"970f66e4-08da-4bff-8533-1e81a9fddbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:40.634156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.605486    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-54a4b2\""
Nov 02 21:32:40.644163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.643307    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699fq\" (UniqueName: \"kubernetes.io/projected/970f66e4-08da-4bff-8533-1e81a9fddbb6-kube-api-access-699fq\") pod \"payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj\" (UID: \"970f66e4-08da-4bff-8533-1e81a9fddbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:40.644163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.643419    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970f66e4-08da-4bff-8533-1e81a9fddbb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj\" (UID: \"970f66e4-08da-4bff-8533-1e81a9fddbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:40.648659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.644377    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970f66e4-08da-4bff-8533-1e81a9fddbb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj\" (UID: \"970f66e4-08da-4bff-8533-1e81a9fddbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:40.911534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.910224    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"]
Nov 02 21:32:40.915558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.915535    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:40.920759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.920727    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-699fq\" (UniqueName: \"kubernetes.io/projected/970f66e4-08da-4bff-8533-1e81a9fddbb6-kube-api-access-699fq\") pod \"payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj\" (UID: \"970f66e4-08da-4bff-8533-1e81a9fddbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:40.929165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.929137    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"]
Nov 02 21:32:40.941506 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.941481    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"]
Nov 02 21:32:40.942042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.942025    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"]
Nov 02 21:32:40.942213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.941556    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:40.984991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.984939    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21279019-ae8a-42a2-b6c0-691809a777ea-plugins-config-volume\") pod \"payload-processing-e2e-api-b-54a4b2-68d494c969-994gk\" (UID: \"21279019-ae8a-42a2-b6c0-691809a777ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:40.985154 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:40.985132    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5f7\" (UniqueName: \"kubernetes.io/projected/21279019-ae8a-42a2-b6c0-691809a777ea-kube-api-access-ld5f7\") pod \"payload-processing-e2e-api-b-54a4b2-68d494c969-994gk\" (UID: \"21279019-ae8a-42a2-b6c0-691809a777ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:41.086496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.086453    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.086496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.086503    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.086775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.086528    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.086775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.086602    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-426t9\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-kube-api-access-426t9\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.086775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.086676    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.086948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.086789    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.086948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.086912    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.087060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.086963    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21279019-ae8a-42a2-b6c0-691809a777ea-plugins-config-volume\") pod \"payload-processing-e2e-api-b-54a4b2-68d494c969-994gk\" (UID: \"21279019-ae8a-42a2-b6c0-691809a777ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:41.087060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.087005    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.087272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.087253    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.087338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.087282    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5f7\" (UniqueName: \"kubernetes.io/projected/21279019-ae8a-42a2-b6c0-691809a777ea-kube-api-access-ld5f7\") pod \"payload-processing-e2e-api-b-54a4b2-68d494c969-994gk\" (UID: \"21279019-ae8a-42a2-b6c0-691809a777ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:41.087951 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.087925    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21279019-ae8a-42a2-b6c0-691809a777ea-plugins-config-volume\") pod \"payload-processing-e2e-api-b-54a4b2-68d494c969-994gk\" (UID: \"21279019-ae8a-42a2-b6c0-691809a777ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:41.138342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.138312    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:41.149839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.148246    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5f7\" (UniqueName: \"kubernetes.io/projected/21279019-ae8a-42a2-b6c0-691809a777ea-kube-api-access-ld5f7\") pod \"payload-processing-e2e-api-b-54a4b2-68d494c969-994gk\" (UID: \"21279019-ae8a-42a2-b6c0-691809a777ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:41.189807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.189692    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.189959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.189822    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.189959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.189849    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.189968    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-426t9\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-kube-api-access-426t9\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190036    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190134    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190178    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190209    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190305    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190407    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190477 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190422    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190477 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190467    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190906 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190670    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.190906 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.190862    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.191337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.191155    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:32:41.192550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.192407    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:32:41.193212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.193170    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.193483 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.193461    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.247711 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.247678    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.248102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.247807    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-426t9\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-kube-api-access-426t9\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.251684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.251662    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:41.259783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.259747    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:41.521675 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:32:41.521648    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970f66e4_08da_4bff_8533_1e81a9fddbb6.slice/crio-36496584eefe71d9a91429d941a5c3ca008183d10a56cf2c8f1ef8604ea1e520 WatchSource:0}: Error finding container 36496584eefe71d9a91429d941a5c3ca008183d10a56cf2c8f1ef8604ea1e520: Status 404 returned error can't find the container with id 36496584eefe71d9a91429d941a5c3ca008183d10a56cf2c8f1ef8604ea1e520
Nov 02 21:32:41.524149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.524125    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"]
Nov 02 21:32:41.627823 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:32:41.627793    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21279019_ae8a_42a2_b6c0_691809a777ea.slice/crio-6562eca3b5f43f70258c012f19ddcd95fd8db69bcf1a4188ee168632534d962d WatchSource:0}: Error finding container 6562eca3b5f43f70258c012f19ddcd95fd8db69bcf1a4188ee168632534d962d: Status 404 returned error can't find the container with id 6562eca3b5f43f70258c012f19ddcd95fd8db69bcf1a4188ee168632534d962d
Nov 02 21:32:41.632545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.632519    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"]
Nov 02 21:32:41.678775 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:32:41.678752    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9df425_9d79_4cf9_8c80_38b4a114aba6.slice/crio-e9b26c5ce22d603e03c5734313a8d5c916c8899ac7f7e03a06971ae3f59dcc36 WatchSource:0}: Error finding container e9b26c5ce22d603e03c5734313a8d5c916c8899ac7f7e03a06971ae3f59dcc36: Status 404 returned error can't find the container with id e9b26c5ce22d603e03c5734313a8d5c916c8899ac7f7e03a06971ae3f59dcc36
Nov 02 21:32:41.681745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.681711    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:41.681851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.681792    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:41.681851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.681835    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:41.683407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.683384    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"]
Nov 02 21:32:41.904600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.904506    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" event={"ID":"9c9df425-9d79-4cf9-8c80-38b4a114aba6","Type":"ContainerStarted","Data":"e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa"}
Nov 02 21:32:41.904600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.904569    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" event={"ID":"9c9df425-9d79-4cf9-8c80-38b4a114aba6","Type":"ContainerStarted","Data":"e9b26c5ce22d603e03c5734313a8d5c916c8899ac7f7e03a06971ae3f59dcc36"}
Nov 02 21:32:41.906553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.906509    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj" event={"ID":"970f66e4-08da-4bff-8533-1e81a9fddbb6","Type":"ContainerStarted","Data":"5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa"}
Nov 02 21:32:41.906553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.906547    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj" event={"ID":"970f66e4-08da-4bff-8533-1e81a9fddbb6","Type":"ContainerStarted","Data":"36496584eefe71d9a91429d941a5c3ca008183d10a56cf2c8f1ef8604ea1e520"}
Nov 02 21:32:41.906764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.906573    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:41.908282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.908260    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk" event={"ID":"21279019-ae8a-42a2-b6c0-691809a777ea","Type":"ContainerStarted","Data":"114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0"}
Nov 02 21:32:41.908377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.908307    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:41.908377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:41.908331    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk" event={"ID":"21279019-ae8a-42a2-b6c0-691809a777ea","Type":"ContainerStarted","Data":"6562eca3b5f43f70258c012f19ddcd95fd8db69bcf1a4188ee168632534d962d"}
Nov 02 21:32:42.022603 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:42.022557    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" podStartSLOduration=2.022542759 podStartE2EDuration="2.022542759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:42.020293606 +0000 UTC m=+1449.283829923" watchObservedRunningTime="2026-11-02 21:32:42.022542759 +0000 UTC m=+1449.286079146"
Nov 02 21:32:42.134645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:42.134587    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk" podStartSLOduration=2.134568452 podStartE2EDuration="2.134568452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:42.133196586 +0000 UTC m=+1449.396732899" watchObservedRunningTime="2026-11-02 21:32:42.134568452 +0000 UTC m=+1449.398104770"
Nov 02 21:32:42.259837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:42.259707    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj" podStartSLOduration=2.259686308 podStartE2EDuration="2.259686308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:42.254860184 +0000 UTC m=+1449.518396501" watchObservedRunningTime="2026-11-02 21:32:42.259686308 +0000 UTC m=+1449.523222626"
Nov 02 21:32:42.261699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:42.261667    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:42.265076 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:42.265047    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 02 21:32:42.265352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:42.265326    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 02 21:32:42.613098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:42.613050    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:32:43.261039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:43.261002    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 02 21:32:43.261463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:43.261058    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 02 21:32:44.151134 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:32:44.149620    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:32:44.267134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:44.264017    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 02 21:32:44.267134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:44.264091    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 02 21:32:45.260385 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:45.260228    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 02 21:32:45.260385 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:45.260315    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 02 21:32:46.260843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:46.260811    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 02 21:32:46.261293 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:46.260871    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 02 21:32:47.260030 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:47.259992    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 02 21:32:47.260221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:47.260048    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 02 21:32:48.265272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:48.265240    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:48.265726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:48.265701    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:48.266409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:48.266390    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:32:51.500167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.500117    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"]
Nov 02 21:32:51.505935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.505912    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.560563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.558849    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bd96ef-openshift-default-dockercfg-wmlwj\""
Nov 02 21:32:51.587100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.587064    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-istio-token\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.587303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.587125    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-data\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.587303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.587217    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-socket\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.587303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.587257    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8bd196-f730-46cd-aa00-23352f984e61-istiod-ca-cert\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.587303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.587294    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-credential-socket\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.587823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.587522    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpvq\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-kube-api-access-8tpvq\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.587823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.587573    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-envoy\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.587823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.587681    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8bd196-f730-46cd-aa00-23352f984e61-istio-podinfo\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.587823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.587733    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-certs\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.616276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.616239    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"]
Nov 02 21:32:51.762185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.757509    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8bd196-f730-46cd-aa00-23352f984e61-istio-podinfo\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.770134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.762457    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-certs\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.787134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.779265    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-istio-token\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.815131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.799347    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-data\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.823133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.815426    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-socket\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.865131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.823513    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-data\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.865131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.802611    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-certs\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.865131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.825534    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-socket\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.881132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.826658    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8bd196-f730-46cd-aa00-23352f984e61-istio-podinfo\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.881132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.875369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8bd196-f730-46cd-aa00-23352f984e61-istiod-ca-cert\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.881132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.876809    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-istio-token\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.901133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.899453    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8bd196-f730-46cd-aa00-23352f984e61-istiod-ca-cert\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.901133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.899646    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-credential-socket\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.901133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.899909    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpvq\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-kube-api-access-8tpvq\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.901133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.899960    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-envoy\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.926185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.904469    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-credential-socket\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.926185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.907560    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-envoy\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:51.986342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:51.985682    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpvq\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-kube-api-access-8tpvq\") pod \"e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") " pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:52.120760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:52.120672    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:52.645668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:52.645633    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"]
Nov 02 21:32:52.678195 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:32:52.678149    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8bd196_f730_46cd_aa00_23352f984e61.slice/crio-feaf1c6e442aedf71971a6b3854094d68dd1f352042ac8212ebe5d5e527b82d0 WatchSource:0}: Error finding container feaf1c6e442aedf71971a6b3854094d68dd1f352042ac8212ebe5d5e527b82d0: Status 404 returned error can't find the container with id feaf1c6e442aedf71971a6b3854094d68dd1f352042ac8212ebe5d5e527b82d0
Nov 02 21:32:52.680777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:52.680740    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:52.680887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:52.680824    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:52.680887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:52.680870    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:32:52.940695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:52.940618    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:32:52.940859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:52.940797    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:32:53.023552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:53.023517    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" event={"ID":"9d8bd196-f730-46cd-aa00-23352f984e61","Type":"ContainerStarted","Data":"1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545"}
Nov 02 21:32:53.023719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:53.023568    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" event={"ID":"9d8bd196-f730-46cd-aa00-23352f984e61","Type":"ContainerStarted","Data":"feaf1c6e442aedf71971a6b3854094d68dd1f352042ac8212ebe5d5e527b82d0"}
Nov 02 21:32:53.120877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:53.120850    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:53.126354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:53.126331    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:53.314654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:53.314599    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" podStartSLOduration=2.314585503 podStartE2EDuration="2.314585503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:53.309575156 +0000 UTC m=+1460.573111497" watchObservedRunningTime="2026-11-02 21:32:53.314585503 +0000 UTC m=+1460.578121819"
Nov 02 21:32:54.030695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:54.030660    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:54.031641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:54.031621    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:32:59.996744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:32:59.996702    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"]
Nov 02 21:33:00.002426 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.002395    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:00.004442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.004417    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"]
Nov 02 21:33:00.103399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.103364    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ffs\" (UniqueName: \"kubernetes.io/projected/04442c1e-13fb-4c6d-af78-72e266feed42-kube-api-access-r6ffs\") pod \"payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx\" (UID: \"04442c1e-13fb-4c6d-af78-72e266feed42\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:00.103594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.103572    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04442c1e-13fb-4c6d-af78-72e266feed42-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx\" (UID: \"04442c1e-13fb-4c6d-af78-72e266feed42\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:00.107490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.107471    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bd96ef\""
Nov 02 21:33:00.107569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.107534    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bd96ef-dockercfg-txxf8\""
Nov 02 21:33:00.204738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.204708    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ffs\" (UniqueName: \"kubernetes.io/projected/04442c1e-13fb-4c6d-af78-72e266feed42-kube-api-access-r6ffs\") pod \"payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx\" (UID: \"04442c1e-13fb-4c6d-af78-72e266feed42\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:00.204924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.204848    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04442c1e-13fb-4c6d-af78-72e266feed42-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx\" (UID: \"04442c1e-13fb-4c6d-af78-72e266feed42\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:00.205764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.205735    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04442c1e-13fb-4c6d-af78-72e266feed42-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx\" (UID: \"04442c1e-13fb-4c6d-af78-72e266feed42\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:00.262674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.262649    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ffs\" (UniqueName: \"kubernetes.io/projected/04442c1e-13fb-4c6d-af78-72e266feed42-kube-api-access-r6ffs\") pod \"payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx\" (UID: \"04442c1e-13fb-4c6d-af78-72e266feed42\") " pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:00.322433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.322406    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:00.693774 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:33:00.693737    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04442c1e_13fb_4c6d_af78_72e266feed42.slice/crio-16d058298d59472d051705309ce4fe1b63384aaaf1b27b8a7d7bf8355e95f76f WatchSource:0}: Error finding container 16d058298d59472d051705309ce4fe1b63384aaaf1b27b8a7d7bf8355e95f76f: Status 404 returned error can't find the container with id 16d058298d59472d051705309ce4fe1b63384aaaf1b27b8a7d7bf8355e95f76f
Nov 02 21:33:00.698709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:00.698683    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"]
Nov 02 21:33:01.080048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:01.080017    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx" event={"ID":"04442c1e-13fb-4c6d-af78-72e266feed42","Type":"ContainerStarted","Data":"8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b"}
Nov 02 21:33:01.080471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:01.080068    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx" event={"ID":"04442c1e-13fb-4c6d-af78-72e266feed42","Type":"ContainerStarted","Data":"16d058298d59472d051705309ce4fe1b63384aaaf1b27b8a7d7bf8355e95f76f"}
Nov 02 21:33:01.080471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:01.080122    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:01.197558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:01.197470    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx" podStartSLOduration=2.197456376 podStartE2EDuration="2.197456376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:01.194804169 +0000 UTC m=+1468.458340511" watchObservedRunningTime="2026-11-02 21:33:01.197456376 +0000 UTC m=+1468.460992694"
Nov 02 21:33:09.338134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.337496    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"]
Nov 02 21:33:09.354154 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.350483    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.440153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.439958    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"]
Nov 02 21:33:09.491136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.488457    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.491136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.488544    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.491136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.488593    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.491136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.488703    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.491136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.488759    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e7b2fb-8608-4235-aea2-b60c1e295270-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.491136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.488806    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.491136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.488882    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.491136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.488928    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n28j\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-kube-api-access-4n28j\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.491136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.488982    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.658103    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.658223    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.658267    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.658661    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.658812    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.658869    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e7b2fb-8608-4235-aea2-b60c1e295270-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.658911    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.658980    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.659015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n28j\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-kube-api-access-4n28j\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.659049    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.659428    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.660017    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e7b2fb-8608-4235-aea2-b60c1e295270-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.660891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.660250    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.661371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.660992    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.684088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.684055    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.699193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.695886    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.718898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.718803    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.720087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.720056    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n28j\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-kube-api-access-4n28j\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:09.980770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:09.980689    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:10.085861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:10.085153    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"]
Nov 02 21:33:10.100642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:10.092562    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51483b5-2nq9b"]
Nov 02 21:33:10.414458 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:33:10.414426    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e7b2fb_8608_4235_aea2_b60c1e295270.slice/crio-29c9b9195413f284795dffffdfab21b17f29abcfa5d998c701c1f10ff7e431a5 WatchSource:0}: Error finding container 29c9b9195413f284795dffffdfab21b17f29abcfa5d998c701c1f10ff7e431a5: Status 404 returned error can't find the container with id 29c9b9195413f284795dffffdfab21b17f29abcfa5d998c701c1f10ff7e431a5
Nov 02 21:33:10.418150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:10.418091    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"]
Nov 02 21:33:10.418466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:10.418425    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:33:10.418551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:10.418515    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:33:10.418674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:10.418566    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:33:10.914546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:10.914510    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc23b07-2906-4a12-b45a-7064e832aedc" path="/var/lib/kubelet/pods/efc23b07-2906-4a12-b45a-7064e832aedc/volumes"
Nov 02 21:33:11.181601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:11.181522    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" event={"ID":"49e7b2fb-8608-4235-aea2-b60c1e295270","Type":"ContainerStarted","Data":"bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4"}
Nov 02 21:33:11.181601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:11.181579    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" event={"ID":"49e7b2fb-8608-4235-aea2-b60c1e295270","Type":"ContainerStarted","Data":"29c9b9195413f284795dffffdfab21b17f29abcfa5d998c701c1f10ff7e431a5"}
Nov 02 21:33:11.352435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:11.352385    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" podStartSLOduration=2.352368771 podStartE2EDuration="2.352368771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:11.347914099 +0000 UTC m=+1478.611450415" watchObservedRunningTime="2026-11-02 21:33:11.352368771 +0000 UTC m=+1478.615905157"
Nov 02 21:33:11.981400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:11.981366    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:11.982596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:11.982562    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 21:33:11.982714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:11.982620    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 21:33:12.090283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:12.090248    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:33:12.980993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:12.980961    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 21:33:12.981194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:12.981009    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 21:33:13.981749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:13.981710    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 21:33:13.982183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:13.981772    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 21:33:14.136966 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:33:14.136901    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:33:14.744990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.744891    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"]
Nov 02 21:33:14.750666 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.750647    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:14.828135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.825773    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"]
Nov 02 21:33:14.846699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.846671    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:33:14.846916 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.846691    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:33:14.846916 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.846764    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:33:14.859143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.859103    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcz2z\" (UniqueName: \"kubernetes.io/projected/c7c754c3-f2d5-4521-97a4-00b039d19756-kube-api-access-mcz2z\") pod \"maas-api-revoke-sub-57442d13-jbzjn\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:14.859263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.859157    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57442d13-jbzjn\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:14.859355 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.859340    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57442d13-jbzjn\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:14.874561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.874542    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:33:14.960762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.960733    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcz2z\" (UniqueName: \"kubernetes.io/projected/c7c754c3-f2d5-4521-97a4-00b039d19756-kube-api-access-mcz2z\") pod \"maas-api-revoke-sub-57442d13-jbzjn\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:14.960938 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.960768    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57442d13-jbzjn\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:14.960938 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.960850    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57442d13-jbzjn\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:14.961530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.961509    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57442d13-jbzjn\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:14.963208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.963189    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57442d13-jbzjn\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:14.981560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.981534    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 21:33:14.981699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:14.981594    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 21:33:15.026992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:15.026954    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcz2z\" (UniqueName: \"kubernetes.io/projected/c7c754c3-f2d5-4521-97a4-00b039d19756-kube-api-access-mcz2z\") pod \"maas-api-revoke-sub-57442d13-jbzjn\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:15.065351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:15.065320    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:15.493843 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:33:15.493811    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c754c3_f2d5_4521_97a4_00b039d19756.slice/crio-c0e9386bb421dbe3bdf364405aa2a36817b69dfbdf74cef1e476062515e78e61 WatchSource:0}: Error finding container c0e9386bb421dbe3bdf364405aa2a36817b69dfbdf74cef1e476062515e78e61: Status 404 returned error can't find the container with id c0e9386bb421dbe3bdf364405aa2a36817b69dfbdf74cef1e476062515e78e61
Nov 02 21:33:15.495201 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:15.495175    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"]
Nov 02 21:33:15.981736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:15.981705    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 21:33:15.981868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:15.981752    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 21:33:16.262188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:16.262139    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn" event={"ID":"c7c754c3-f2d5-4521-97a4-00b039d19756","Type":"ContainerStarted","Data":"494c51b1d0b74c71c2572218505d424d48d8d91986b857e79687588e94793c15"}
Nov 02 21:33:16.262657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:16.262206    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn" event={"ID":"c7c754c3-f2d5-4521-97a4-00b039d19756","Type":"ContainerStarted","Data":"c0e9386bb421dbe3bdf364405aa2a36817b69dfbdf74cef1e476062515e78e61"}
Nov 02 21:33:16.386139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:16.386032    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn" podStartSLOduration=2.386013191 podStartE2EDuration="2.386013191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:16.383394299 +0000 UTC m=+1483.646930618" watchObservedRunningTime="2026-11-02 21:33:16.386013191 +0000 UTC m=+1483.649549510"
Nov 02 21:33:16.981826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:16.981738    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 21:33:16.981826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:16.981795    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 21:33:17.304133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:17.303788    4528 generic.go:356] "Generic (PLEG): container finished" podID="c7c754c3-f2d5-4521-97a4-00b039d19756" containerID="494c51b1d0b74c71c2572218505d424d48d8d91986b857e79687588e94793c15" exitCode=0
Nov 02 21:33:17.304133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:17.303864    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn" event={"ID":"c7c754c3-f2d5-4521-97a4-00b039d19756","Type":"ContainerDied","Data":"494c51b1d0b74c71c2572218505d424d48d8d91986b857e79687588e94793c15"}
Nov 02 21:33:17.984959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:17.984901    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:17.985432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:17.985414    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:17.986027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:17.986006    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:33:18.492577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.492551    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:18.611045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.610967    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-openshift-ca\") pod \"c7c754c3-f2d5-4521-97a4-00b039d19756\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") "
Nov 02 21:33:18.611045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.611021    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-xks-ca\") pod \"c7c754c3-f2d5-4521-97a4-00b039d19756\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") "
Nov 02 21:33:18.611273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.611141    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c754c3-f2d5-4521-97a4-00b039d19756-kube-api-access-mcz2z\" (UniqueName: \"kubernetes.io/projected/c7c754c3-f2d5-4521-97a4-00b039d19756-kube-api-access-mcz2z\") pod \"c7c754c3-f2d5-4521-97a4-00b039d19756\" (UID: \"c7c754c3-f2d5-4521-97a4-00b039d19756\") "
Nov 02 21:33:18.611358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.611335    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-openshift-ca" pod "c7c754c3-f2d5-4521-97a4-00b039d19756" (UID: "c7c754c3-f2d5-4521-97a4-00b039d19756"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:33:18.611805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.611787    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:33:18.614344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.614321    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-xks-ca" pod "c7c754c3-f2d5-4521-97a4-00b039d19756" (UID: "c7c754c3-f2d5-4521-97a4-00b039d19756"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:33:18.614441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.614412    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c754c3-f2d5-4521-97a4-00b039d19756-kube-api-access-mcz2z" pod "c7c754c3-f2d5-4521-97a4-00b039d19756" (UID: "c7c754c3-f2d5-4521-97a4-00b039d19756"). InnerVolumeSpecName "kube-api-access-mcz2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:33:18.713594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.713561    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c754c3-f2d5-4521-97a4-00b039d19756-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:33:18.713594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:18.713588    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcz2z\" (UniqueName: \"kubernetes.io/projected/c7c754c3-f2d5-4521-97a4-00b039d19756-kube-api-access-mcz2z\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:33:19.320985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:19.320936    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"
Nov 02 21:33:19.321185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:19.320993    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn" event={"ID":"c7c754c3-f2d5-4521-97a4-00b039d19756","Type":"ContainerDied","Data":"c0e9386bb421dbe3bdf364405aa2a36817b69dfbdf74cef1e476062515e78e61"}
Nov 02 21:33:19.321185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:19.321052    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0e9386bb421dbe3bdf364405aa2a36817b69dfbdf74cef1e476062515e78e61"
Nov 02 21:33:24.137615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:24.137584    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:33:25.050015 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:25.049984    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"]
Nov 02 21:33:25.100150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:25.100122    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2044f278-6qhqf"]
Nov 02 21:33:26.914358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:26.914326    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="449b2282-4642-479e-9fcb-b01db91a5697" path="/var/lib/kubelet/pods/449b2282-4642-479e-9fcb-b01db91a5697/volumes"
Nov 02 21:33:32.049664 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:32.049622    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"]
Nov 02 21:33:32.099472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:32.099437    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9345bd2a-z5qs4"]
Nov 02 21:33:32.914773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:32.914741    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11666a4-2dd6-405d-b328-1560dd6812ac" path="/var/lib/kubelet/pods/c11666a4-2dd6-405d-b328-1560dd6812ac/volumes"
Nov 02 21:33:36.012134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.011486    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"]
Nov 02 21:33:36.016130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.012639    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c7c754c3-f2d5-4521-97a4-00b039d19756" containerName="revoke-keys"
Nov 02 21:33:36.016130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.012664    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c754c3-f2d5-4521-97a4-00b039d19756" containerName="revoke-keys"
Nov 02 21:33:36.016130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.012830    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c754c3-f2d5-4521-97a4-00b039d19756" containerName="revoke-keys"
Nov 02 21:33:36.042133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.040565    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:36.112245 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.112213    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-682116\""
Nov 02 21:33:36.112498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.112386    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa666493-4f89-4a72-8a3e-ffc854458af3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-682116-7c5c94c747-6vr46\" (UID: \"aa666493-4f89-4a72-8a3e-ffc854458af3\") " pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:36.113402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.113359    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-682116-dockercfg-9qln9\""
Nov 02 21:33:36.118130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.113429    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f86w8\" (UniqueName: \"kubernetes.io/projected/aa666493-4f89-4a72-8a3e-ffc854458af3-kube-api-access-f86w8\") pod \"payload-processing-e2e-api-a-682116-7c5c94c747-6vr46\" (UID: \"aa666493-4f89-4a72-8a3e-ffc854458af3\") " pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:36.187136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.158450    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"]
Nov 02 21:33:36.187409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.187385    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61fd519-5766n"]
Nov 02 21:33:36.187564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.187550    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"]
Nov 02 21:33:36.235760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.235715    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa666493-4f89-4a72-8a3e-ffc854458af3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-682116-7c5c94c747-6vr46\" (UID: \"aa666493-4f89-4a72-8a3e-ffc854458af3\") " pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:36.236319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.236295    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f86w8\" (UniqueName: \"kubernetes.io/projected/aa666493-4f89-4a72-8a3e-ffc854458af3-kube-api-access-f86w8\") pod \"payload-processing-e2e-api-a-682116-7c5c94c747-6vr46\" (UID: \"aa666493-4f89-4a72-8a3e-ffc854458af3\") " pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:36.238513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.238479    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa666493-4f89-4a72-8a3e-ffc854458af3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-682116-7c5c94c747-6vr46\" (UID: \"aa666493-4f89-4a72-8a3e-ffc854458af3\") " pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:36.295518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.295455    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f86w8\" (UniqueName: \"kubernetes.io/projected/aa666493-4f89-4a72-8a3e-ffc854458af3-kube-api-access-f86w8\") pod \"payload-processing-e2e-api-a-682116-7c5c94c747-6vr46\" (UID: \"aa666493-4f89-4a72-8a3e-ffc854458af3\") " pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:36.459181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.459147    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:36.835624 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:33:36.835591    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa666493_4f89_4a72_8a3e_ffc854458af3.slice/crio-131d57655185b8a648ef860f009adce9ddca5b4803729ba94196b426e8a7f9f5 WatchSource:0}: Error finding container 131d57655185b8a648ef860f009adce9ddca5b4803729ba94196b426e8a7f9f5: Status 404 returned error can't find the container with id 131d57655185b8a648ef860f009adce9ddca5b4803729ba94196b426e8a7f9f5
Nov 02 21:33:36.838974 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.838948    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"]
Nov 02 21:33:36.917363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:36.917335    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="011aac80-b631-471d-8224-b0303bf9b6ff" path="/var/lib/kubelet/pods/011aac80-b631-471d-8224-b0303bf9b6ff/volumes"
Nov 02 21:33:37.469098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:37.469064    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46" event={"ID":"aa666493-4f89-4a72-8a3e-ffc854458af3","Type":"ContainerStarted","Data":"f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0"}
Nov 02 21:33:37.469534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:37.469127    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46" event={"ID":"aa666493-4f89-4a72-8a3e-ffc854458af3","Type":"ContainerStarted","Data":"131d57655185b8a648ef860f009adce9ddca5b4803729ba94196b426e8a7f9f5"}
Nov 02 21:33:37.469534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:37.469152    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:37.583743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:37.583682    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46" podStartSLOduration=2.583667054 podStartE2EDuration="2.583667054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:37.583142064 +0000 UTC m=+1504.846678376" watchObservedRunningTime="2026-11-02 21:33:37.583667054 +0000 UTC m=+1504.847203371"
Nov 02 21:33:44.138185 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:33:44.138133    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:33:45.188501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.188458    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"]
Nov 02 21:33:45.194290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.194271    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.246889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.246861    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:33:45.246889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.246864    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:33:45.294568 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.294536    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbs5x\" (UniqueName: \"kubernetes.io/projected/130fde42-055a-4e04-b040-6b0931caa0e3-kube-api-access-sbs5x\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-cc79s\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.294740 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.294724    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-cc79s\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.294784 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.294759    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-cc79s\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.297258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.297228    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:33:45.297374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.297302    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:33:45.302058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.302032    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"]
Nov 02 21:33:45.396442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.396410    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-cc79s\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.396609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.396452    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-cc79s\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.396609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.396565    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbs5x\" (UniqueName: \"kubernetes.io/projected/130fde42-055a-4e04-b040-6b0931caa0e3-kube-api-access-sbs5x\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-cc79s\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.397259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.397241    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-cc79s\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.398873 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.398853    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-cc79s\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.454623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.454555    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbs5x\" (UniqueName: \"kubernetes.io/projected/130fde42-055a-4e04-b040-6b0931caa0e3-kube-api-access-sbs5x\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-cc79s\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.507790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.507750    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:45.879991 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:33:45.879959    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130fde42_055a_4e04_b040_6b0931caa0e3.slice/crio-bbbad8c60ab6af70a19d2e84b52da1183245e700e52cba69cd46a4d2fa7b6d2d WatchSource:0}: Error finding container bbbad8c60ab6af70a19d2e84b52da1183245e700e52cba69cd46a4d2fa7b6d2d: Status 404 returned error can't find the container with id bbbad8c60ab6af70a19d2e84b52da1183245e700e52cba69cd46a4d2fa7b6d2d
Nov 02 21:33:45.882577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:45.882543    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"]
Nov 02 21:33:46.536263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:46.536222    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s" event={"ID":"130fde42-055a-4e04-b040-6b0931caa0e3","Type":"ContainerStarted","Data":"6a1f11c32e5514d55cfe4acea051d2008d308353e4247972b583e86a100a660e"}
Nov 02 21:33:46.536718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:46.536313    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s" event={"ID":"130fde42-055a-4e04-b040-6b0931caa0e3","Type":"ContainerStarted","Data":"bbbad8c60ab6af70a19d2e84b52da1183245e700e52cba69cd46a4d2fa7b6d2d"}
Nov 02 21:33:46.652649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:46.652596    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s" podStartSLOduration=1.6525834750000001 podStartE2EDuration="1.652583475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:46.649779221 +0000 UTC m=+1513.913315544" watchObservedRunningTime="2026-11-02 21:33:46.652583475 +0000 UTC m=+1513.916119791"
Nov 02 21:33:47.051181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:47.051153    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"]
Nov 02 21:33:47.055148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:47.055124    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef178d2-zmnrs"]
Nov 02 21:33:47.545006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:47.544970    4528 generic.go:356] "Generic (PLEG): container finished" podID="130fde42-055a-4e04-b040-6b0931caa0e3" containerID="6a1f11c32e5514d55cfe4acea051d2008d308353e4247972b583e86a100a660e" exitCode=0
Nov 02 21:33:47.545452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:47.545051    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s" event={"ID":"130fde42-055a-4e04-b040-6b0931caa0e3","Type":"ContainerDied","Data":"6a1f11c32e5514d55cfe4acea051d2008d308353e4247972b583e86a100a660e"}
Nov 02 21:33:48.494132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:48.489232    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:33:48.899623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:48.899598    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:48.915296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:48.915267    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916bdd95-3ab7-47b9-a385-c545e5de6a55" path="/var/lib/kubelet/pods/916bdd95-3ab7-47b9-a385-c545e5de6a55/volumes"
Nov 02 21:33:49.051152 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.051123    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-openshift-ca\") pod \"130fde42-055a-4e04-b040-6b0931caa0e3\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") "
Nov 02 21:33:49.051338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.051295    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-xks-ca\") pod \"130fde42-055a-4e04-b040-6b0931caa0e3\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") "
Nov 02 21:33:49.051533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.051516    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130fde42-055a-4e04-b040-6b0931caa0e3-kube-api-access-sbs5x\" (UniqueName: \"kubernetes.io/projected/130fde42-055a-4e04-b040-6b0931caa0e3-kube-api-access-sbs5x\") pod \"130fde42-055a-4e04-b040-6b0931caa0e3\" (UID: \"130fde42-055a-4e04-b040-6b0931caa0e3\") "
Nov 02 21:33:49.051603 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.051568    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-openshift-ca" pod "130fde42-055a-4e04-b040-6b0931caa0e3" (UID: "130fde42-055a-4e04-b040-6b0931caa0e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:33:49.052404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.052373    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:33:49.054083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.054051    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130fde42-055a-4e04-b040-6b0931caa0e3-kube-api-access-sbs5x" pod "130fde42-055a-4e04-b040-6b0931caa0e3" (UID: "130fde42-055a-4e04-b040-6b0931caa0e3"). InnerVolumeSpecName "kube-api-access-sbs5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:33:49.054083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.054072    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-xks-ca" pod "130fde42-055a-4e04-b040-6b0931caa0e3" (UID: "130fde42-055a-4e04-b040-6b0931caa0e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:33:49.153547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.153516    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbs5x\" (UniqueName: \"kubernetes.io/projected/130fde42-055a-4e04-b040-6b0931caa0e3-kube-api-access-sbs5x\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:33:49.153547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.153544    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/130fde42-055a-4e04-b040-6b0931caa0e3-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:33:49.586086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.586057    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"
Nov 02 21:33:49.586292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.586055    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s" event={"ID":"130fde42-055a-4e04-b040-6b0931caa0e3","Type":"ContainerDied","Data":"bbbad8c60ab6af70a19d2e84b52da1183245e700e52cba69cd46a4d2fa7b6d2d"}
Nov 02 21:33:49.586292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:49.586203    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbbad8c60ab6af70a19d2e84b52da1183245e700e52cba69cd46a4d2fa7b6d2d"
Nov 02 21:33:52.051574 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:52.051540    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"]
Nov 02 21:33:52.101782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:52.101746    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6391aeaa-pgvsj"]
Nov 02 21:33:52.916274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:52.916243    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2f078e-f6ac-4026-80c6-2dea29cc2294" path="/var/lib/kubelet/pods/fb2f078e-f6ac-4026-80c6-2dea29cc2294/volumes"
Nov 02 21:33:56.580467 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:56.580436    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"]
Nov 02 21:33:56.687250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:33:56.687217    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"]
Nov 02 21:34:01.580757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:01.580719    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9" podUID="4a1f5dff-d162-4d80-b70c-04f273bcee9b" containerName="payload-processing" containerID="cri-o://a45b29ac0aca8f5a1057dba8e8dfdef6722adee6fee4dee75e8add3187a39530" gracePeriod=25
Nov 02 21:34:01.687718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:01.687664    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv" podUID="c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" containerName="payload-pre-processing" containerID="cri-o://55cec3fe2ba6388358de158acfa1a72fca6f67c304cb3bbef21646af54ac63f2" gracePeriod=25
Nov 02 21:34:04.050768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:04.050734    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"]
Nov 02 21:34:04.100866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:04.100831    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe882e8-4w884"]
Nov 02 21:34:04.914020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:04.913986    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77218fa-47f1-4a29-9efb-420c1c0df91e" path="/var/lib/kubelet/pods/e77218fa-47f1-4a29-9efb-420c1c0df91e/volumes"
Nov 02 21:34:08.657325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:08.657286    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9" podUID="4a1f5dff-d162-4d80-b70c-04f273bcee9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:34:08.657325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:08.657322    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv" podUID="c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:34:09.048536 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:09.048504    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"]
Nov 02 21:34:09.099427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:09.099389    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bba4f0b-khx8r"]
Nov 02 21:34:10.003306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:10.003276    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:34:11.083224 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:11.083181    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db59701-54b0-4f0c-b534-eeedcf9dc520" path="/var/lib/kubelet/pods/0db59701-54b0-4f0c-b534-eeedcf9dc520/volumes"
Nov 02 21:34:14.141089 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:34:14.141053    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:34:15.051194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:15.051158    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"]
Nov 02 21:34:15.123368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:15.123328    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75845a1a-pddtj"]
Nov 02 21:34:16.916404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:16.916370    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="848c527e-1850-4526-a9bf-034be4084fa9" path="/var/lib/kubelet/pods/848c527e-1850-4526-a9bf-034be4084fa9/volumes"
Nov 02 21:34:18.656893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:18.656856    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv" podUID="c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:34:18.657350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:18.656868    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9" podUID="4a1f5dff-d162-4d80-b70c-04f273bcee9b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:34:20.893194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:20.893095    4528 generic.go:356] "Generic (PLEG): container finished" podID="9d8bd196-f730-46cd-aa00-23352f984e61" containerID="1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545" exitCode=0
Nov 02 21:34:20.893194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:20.893174    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" event={"ID":"9d8bd196-f730-46cd-aa00-23352f984e61","Type":"ContainerDied","Data":"1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545"}
Nov 02 21:34:20.893703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:20.893682    4528 scope.go:122] "RemoveContainer" containerID="1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545"
Nov 02 21:34:20.895239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:20.895206    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:34:20.895354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:20.895290    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:34:20.895354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:20.895338    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:34:21.052829 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.052796    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"]
Nov 02 21:34:21.057158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.057129    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d88ae7-tsrpr"]
Nov 02 21:34:21.934387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.934350    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" event={"ID":"9d8bd196-f730-46cd-aa00-23352f984e61","Type":"ContainerStarted","Data":"7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e"}
Nov 02 21:34:21.936769 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.936736    4528 generic.go:356] "Generic (PLEG): container finished" podID="4a1f5dff-d162-4d80-b70c-04f273bcee9b" containerID="a45b29ac0aca8f5a1057dba8e8dfdef6722adee6fee4dee75e8add3187a39530" exitCode=0
Nov 02 21:34:21.936918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.936808    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9" event={"ID":"4a1f5dff-d162-4d80-b70c-04f273bcee9b","Type":"ContainerDied","Data":"a45b29ac0aca8f5a1057dba8e8dfdef6722adee6fee4dee75e8add3187a39530"}
Nov 02 21:34:21.936918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.936864    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9" event={"ID":"4a1f5dff-d162-4d80-b70c-04f273bcee9b","Type":"ContainerDied","Data":"dbc07437bc6061bde14e6675d3e9355a57b650a5f00192618342a4e918263bcc"}
Nov 02 21:34:21.936918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.936882    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbc07437bc6061bde14e6675d3e9355a57b650a5f00192618342a4e918263bcc"
Nov 02 21:34:21.940195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.940168    4528 generic.go:356] "Generic (PLEG): container finished" podID="c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" containerID="55cec3fe2ba6388358de158acfa1a72fca6f67c304cb3bbef21646af54ac63f2" exitCode=0
Nov 02 21:34:21.940407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.940378    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv" event={"ID":"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2","Type":"ContainerDied","Data":"55cec3fe2ba6388358de158acfa1a72fca6f67c304cb3bbef21646af54ac63f2"}
Nov 02 21:34:21.944134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:21.944096    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:34:22.036637 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.036596    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1f5dff-d162-4d80-b70c-04f273bcee9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f5dff-d162-4d80-b70c-04f273bcee9b-plugins-config-volume\") pod \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\" (UID: \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\") "
Nov 02 21:34:22.037059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.036871    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1f5dff-d162-4d80-b70c-04f273bcee9b-kube-api-access-4p7kt\" (UniqueName: \"kubernetes.io/projected/4a1f5dff-d162-4d80-b70c-04f273bcee9b-kube-api-access-4p7kt\") pod \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\" (UID: \"4a1f5dff-d162-4d80-b70c-04f273bcee9b\") "
Nov 02 21:34:22.037059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.037027    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1f5dff-d162-4d80-b70c-04f273bcee9b-plugins-config-volume" pod "4a1f5dff-d162-4d80-b70c-04f273bcee9b" (UID: "4a1f5dff-d162-4d80-b70c-04f273bcee9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:34:22.037541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.037512    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f5dff-d162-4d80-b70c-04f273bcee9b-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:34:22.039339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.039316    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1f5dff-d162-4d80-b70c-04f273bcee9b-kube-api-access-4p7kt" pod "4a1f5dff-d162-4d80-b70c-04f273bcee9b" (UID: "4a1f5dff-d162-4d80-b70c-04f273bcee9b"). InnerVolumeSpecName "kube-api-access-4p7kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:34:22.071464 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.071441    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:34:22.127194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.123504    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:34:22.127194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.127071    4528 patch_prober.go:29] interesting pod/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 21:34:22.127194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.127147    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 21:34:22.138532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.138498    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-plugins-config-volume\") pod \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\" (UID: \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\") "
Nov 02 21:34:22.138834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.138815    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-kube-api-access-n8f5w\" (UniqueName: \"kubernetes.io/projected/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-kube-api-access-n8f5w\") pod \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\" (UID: \"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2\") "
Nov 02 21:34:22.138925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.138906    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-plugins-config-volume" pod "c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" (UID: "c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:34:22.139435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.139412    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p7kt\" (UniqueName: \"kubernetes.io/projected/4a1f5dff-d162-4d80-b70c-04f273bcee9b-kube-api-access-4p7kt\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:34:22.139555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.139437    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:34:22.141011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.140987    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-kube-api-access-n8f5w" pod "c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" (UID: "c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2"). InnerVolumeSpecName "kube-api-access-n8f5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:34:22.241298 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.241263    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8f5w\" (UniqueName: \"kubernetes.io/projected/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2-kube-api-access-n8f5w\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:34:22.933300 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.933268    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958fc55b-d0c8-47be-9c66-146f4570103a" path="/var/lib/kubelet/pods/958fc55b-d0c8-47be-9c66-146f4570103a/volumes"
Nov 02 21:34:22.948922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.948888    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"
Nov 02 21:34:22.949351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.948916    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv" event={"ID":"c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2","Type":"ContainerDied","Data":"8b228d25edd7fe81ed9bda0bc10f0f7e0bfd405fb9a41a6f745a96ed275a0c28"}
Nov 02 21:34:22.949351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.948998    4528 scope.go:122] "RemoveContainer" containerID="55cec3fe2ba6388358de158acfa1a72fca6f67c304cb3bbef21646af54ac63f2"
Nov 02 21:34:22.949351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:22.949098    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"
Nov 02 21:34:23.124132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:23.123554    4528 patch_prober.go:29] interesting pod/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 21:34:23.124132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:23.123619    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 21:34:23.138252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:23.138210    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"]
Nov 02 21:34:23.187837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:23.187805    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-vs5c9"]
Nov 02 21:34:23.300853 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:23.300819    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"]
Nov 02 21:34:23.329613 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:23.329564    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-zs9xv"]
Nov 02 21:34:24.031477 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:24.031449    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:34:24.127133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:24.126523    4528 patch_prober.go:29] interesting pod/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 21:34:24.127133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:24.126585    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 21:34:24.932508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:24.932478    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1f5dff-d162-4d80-b70c-04f273bcee9b" path="/var/lib/kubelet/pods/4a1f5dff-d162-4d80-b70c-04f273bcee9b/volumes"
Nov 02 21:34:24.932889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:24.932875    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" path="/var/lib/kubelet/pods/c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2/volumes"
Nov 02 21:34:25.123132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:25.122526    4528 patch_prober.go:29] interesting pod/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 21:34:25.123132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:25.122587    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 21:34:26.123131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:26.122529    4528 patch_prober.go:29] interesting pod/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 21:34:26.123131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:26.122592    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 21:34:27.125901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:27.125870    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:34:27.127026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:27.127004    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:34:33.386294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:33.386259    4528 scope.go:122] "RemoveContainer" containerID="c007c7a14e501ceee38b8051b479ef71760577bd4c7fb3948847f1ca9eb35c72"
Nov 02 21:34:33.421481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:33.420839    4528 scope.go:122] "RemoveContainer" containerID="4efddffeb0c5fe98e5b9528174fc7bdd6ea08e5d7d5464f487c6777f2229ed64"
Nov 02 21:34:33.470473 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:33.469136    4528 scope.go:122] "RemoveContainer" containerID="c50391be9baf6ffe6720eadfd2a9cd9222976cecb1542401871a9a73538be1a7"
Nov 02 21:34:33.495377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:33.495353    4528 scope.go:122] "RemoveContainer" containerID="3e81694d3a2a3317e5fab489347a0ec198baeab9ca6642ade33b46a7bc5cad26"
Nov 02 21:34:36.874243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:36.874209    4528 patch_prober.go:29] interesting pod/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:34:36.874651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:36.874285    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:34:37.049692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:37.049661    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"]
Nov 02 21:34:37.099928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:37.099892    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad87a75-557sz"]
Nov 02 21:34:38.090485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:38.090408    4528 generic.go:356] "Generic (PLEG): container finished" podID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerID="3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7" exitCode=0
Nov 02 21:34:38.090974 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:38.090483    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" event={"ID":"dbfd76d5-b34a-41e6-badb-43f8a4a9c474","Type":"ContainerDied","Data":"3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7"}
Nov 02 21:34:38.090974 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:38.090933    4528 scope.go:122] "RemoveContainer" containerID="3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7"
Nov 02 21:34:38.092538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:38.092508    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:34:38.092651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:38.092591    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:34:38.092651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:38.092624    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:34:38.933569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:38.933535    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7d3cff-9468-421b-acdd-812f2f79c48b" path="/var/lib/kubelet/pods/4b7d3cff-9468-421b-acdd-812f2f79c48b/volumes"
Nov 02 21:34:39.053062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053030    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"]
Nov 02 21:34:39.053679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053665    4528 cpu_manager.go:414] "containerMap: removing container" podUID="130fde42-055a-4e04-b040-6b0931caa0e3" containerName="revoke-keys"
Nov 02 21:34:39.053720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053681    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130fde42-055a-4e04-b040-6b0931caa0e3" containerName="revoke-keys"
Nov 02 21:34:39.053720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053698    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4a1f5dff-d162-4d80-b70c-04f273bcee9b" containerName="payload-processing"
Nov 02 21:34:39.053720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053703    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1f5dff-d162-4d80-b70c-04f273bcee9b" containerName="payload-processing"
Nov 02 21:34:39.053817 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053737    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" containerName="payload-pre-processing"
Nov 02 21:34:39.053817 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053741    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" containerName="payload-pre-processing"
Nov 02 21:34:39.053884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053835    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bfb456-7eb0-4fb0-8a1f-51fcfc7294c2" containerName="payload-pre-processing"
Nov 02 21:34:39.053884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053846    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="130fde42-055a-4e04-b040-6b0931caa0e3" containerName="revoke-keys"
Nov 02 21:34:39.053884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.053853    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1f5dff-d162-4d80-b70c-04f273bcee9b" containerName="payload-processing"
Nov 02 21:34:39.057584 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.057569    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.098802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.098762    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" event={"ID":"dbfd76d5-b34a-41e6-badb-43f8a4a9c474","Type":"ContainerStarted","Data":"3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c"}
Nov 02 21:34:39.130682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.130645    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"]
Nov 02 21:34:39.171767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.171736    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:34:39.171988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.171957    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:34:39.171988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.171978    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:34:39.172081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.172003    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:34:39.233412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.233319    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2a5ff8-8jqfv\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.233412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.233367    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2a5ff8-8jqfv\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.233744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.233724    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfqqp\" (UniqueName: \"kubernetes.io/projected/e35bf9f3-4deb-417e-a3cb-7facb777aa87-kube-api-access-sfqqp\") pod \"maas-api-revoke-sub-3f2a5ff8-8jqfv\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.334980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.334942    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfqqp\" (UniqueName: \"kubernetes.io/projected/e35bf9f3-4deb-417e-a3cb-7facb777aa87-kube-api-access-sfqqp\") pod \"maas-api-revoke-sub-3f2a5ff8-8jqfv\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.335230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.335078    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2a5ff8-8jqfv\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.335230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.335145    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2a5ff8-8jqfv\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.335753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.335722    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2a5ff8-8jqfv\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.337454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.337434    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2a5ff8-8jqfv\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.392117 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.392075    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfqqp\" (UniqueName: \"kubernetes.io/projected/e35bf9f3-4deb-417e-a3cb-7facb777aa87-kube-api-access-sfqqp\") pod \"maas-api-revoke-sub-3f2a5ff8-8jqfv\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.670691 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.670662    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:39.970666 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.970532    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:34:39.982286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.971709    4528 patch_prober.go:29] interesting pod/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 21:34:39.982286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:39.971759    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 21:34:40.100500 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:34:40.100471    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35bf9f3_4deb_417e_a3cb_7facb777aa87.slice/crio-77d288baccbd8c7fb6fe70797027d417e2d3cb0be7943200087349774b1d8e4a WatchSource:0}: Error finding container 77d288baccbd8c7fb6fe70797027d417e2d3cb0be7943200087349774b1d8e4a: Status 404 returned error can't find the container with id 77d288baccbd8c7fb6fe70797027d417e2d3cb0be7943200087349774b1d8e4a
Nov 02 21:34:40.131059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:40.131034    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"]
Nov 02 21:34:40.968164 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:40.968106    4528 patch_prober.go:29] interesting pod/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 21:34:40.968331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:40.968182    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 21:34:41.141614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:41.141572    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv" event={"ID":"e35bf9f3-4deb-417e-a3cb-7facb777aa87","Type":"ContainerStarted","Data":"d617797cbc9762b5f1ec9ee97531ed9cf666614e73e943abc144d2ef73a97e83"}
Nov 02 21:34:41.142027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:41.141641    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv" event={"ID":"e35bf9f3-4deb-417e-a3cb-7facb777aa87","Type":"ContainerStarted","Data":"77d288baccbd8c7fb6fe70797027d417e2d3cb0be7943200087349774b1d8e4a"}
Nov 02 21:34:41.260661 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:41.260611    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv" podStartSLOduration=2.260597429 podStartE2EDuration="2.260597429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:41.255366139 +0000 UTC m=+1568.518902458" watchObservedRunningTime="2026-11-02 21:34:41.260597429 +0000 UTC m=+1568.524133745"
Nov 02 21:34:41.968633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:41.968597    4528 patch_prober.go:29] interesting pod/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 21:34:41.968803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:41.968653    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 21:34:42.152702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:42.152668    4528 generic.go:356] "Generic (PLEG): container finished" podID="e35bf9f3-4deb-417e-a3cb-7facb777aa87" containerID="d617797cbc9762b5f1ec9ee97531ed9cf666614e73e943abc144d2ef73a97e83" exitCode=0
Nov 02 21:34:42.153138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:42.152736    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv" event={"ID":"e35bf9f3-4deb-417e-a3cb-7facb777aa87","Type":"ContainerDied","Data":"d617797cbc9762b5f1ec9ee97531ed9cf666614e73e943abc144d2ef73a97e83"}
Nov 02 21:34:42.968719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:42.968689    4528 patch_prober.go:29] interesting pod/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 21:34:42.968870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:42.968734    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 21:34:43.049235 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.049195    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"]
Nov 02 21:34:43.053135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.053092    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435eae3f-9bxgt"]
Nov 02 21:34:43.213430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.213397    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"]
Nov 02 21:34:43.382833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.382559    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:43.463354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.463327    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35bf9f3-4deb-417e-a3cb-7facb777aa87-kube-api-access-sfqqp\" (UniqueName: \"kubernetes.io/projected/e35bf9f3-4deb-417e-a3cb-7facb777aa87-kube-api-access-sfqqp\") pod \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") "
Nov 02 21:34:43.463497 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.463414    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-openshift-ca\") pod \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") "
Nov 02 21:34:43.463497 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.463487    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-xks-ca\") pod \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\" (UID: \"e35bf9f3-4deb-417e-a3cb-7facb777aa87\") "
Nov 02 21:34:43.463806 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.463782    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-openshift-ca" pod "e35bf9f3-4deb-417e-a3cb-7facb777aa87" (UID: "e35bf9f3-4deb-417e-a3cb-7facb777aa87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:34:43.464052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.464037    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:34:43.465369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.465339    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35bf9f3-4deb-417e-a3cb-7facb777aa87-kube-api-access-sfqqp" pod "e35bf9f3-4deb-417e-a3cb-7facb777aa87" (UID: "e35bf9f3-4deb-417e-a3cb-7facb777aa87"). InnerVolumeSpecName "kube-api-access-sfqqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:34:43.465464 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.465438    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-xks-ca" pod "e35bf9f3-4deb-417e-a3cb-7facb777aa87" (UID: "e35bf9f3-4deb-417e-a3cb-7facb777aa87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:34:43.579939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.579857    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfqqp\" (UniqueName: \"kubernetes.io/projected/e35bf9f3-4deb-417e-a3cb-7facb777aa87-kube-api-access-sfqqp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:34:43.579939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.579901    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35bf9f3-4deb-417e-a3cb-7facb777aa87-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:34:43.969064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.968983    4528 patch_prober.go:29] interesting pod/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 21:34:43.969064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:43.969050    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 21:34:44.114086 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:34:44.114050    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:34:44.168281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:44.168247    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"
Nov 02 21:34:44.168476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:44.168295    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv" event={"ID":"e35bf9f3-4deb-417e-a3cb-7facb777aa87","Type":"ContainerDied","Data":"77d288baccbd8c7fb6fe70797027d417e2d3cb0be7943200087349774b1d8e4a"}
Nov 02 21:34:44.168476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:44.168355    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77d288baccbd8c7fb6fe70797027d417e2d3cb0be7943200087349774b1d8e4a"
Nov 02 21:34:44.966864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:44.966770    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af239596-ac2a-4136-97d2-db32ad4808ec" path="/var/lib/kubelet/pods/af239596-ac2a-4136-97d2-db32ad4808ec/volumes"
Nov 02 21:34:45.000165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:44.996727    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:34:45.000165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:44.998357    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:34:45.005132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:45.004303    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:34:47.052581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:47.052547    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"]
Nov 02 21:34:47.102946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:47.102914    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3408d1-58thw"]
Nov 02 21:34:48.050565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.050534    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"]
Nov 02 21:34:48.101470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.101438    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16a2f95-mmdmf"]
Nov 02 21:34:48.215522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.215457    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj" podUID="9b0e06a0-bc8e-4cb5-b81c-b85942339178" containerName="payload-pre-processing" containerID="cri-o://e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926" gracePeriod=25
Nov 02 21:34:48.496859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.496834    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:34:48.609024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.608922    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0e06a0-bc8e-4cb5-b81c-b85942339178-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e06a0-bc8e-4cb5-b81c-b85942339178-plugins-config-volume\") pod \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\" (UID: \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\") "
Nov 02 21:34:48.609485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.609023    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0e06a0-bc8e-4cb5-b81c-b85942339178-kube-api-access-84d5c\" (UniqueName: \"kubernetes.io/projected/9b0e06a0-bc8e-4cb5-b81c-b85942339178-kube-api-access-84d5c\") pod \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\" (UID: \"9b0e06a0-bc8e-4cb5-b81c-b85942339178\") "
Nov 02 21:34:48.609485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.609413    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e06a0-bc8e-4cb5-b81c-b85942339178-plugins-config-volume" pod "9b0e06a0-bc8e-4cb5-b81c-b85942339178" (UID: "9b0e06a0-bc8e-4cb5-b81c-b85942339178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:34:48.609890 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.609870    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e06a0-bc8e-4cb5-b81c-b85942339178-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:34:48.611466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.611430    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0e06a0-bc8e-4cb5-b81c-b85942339178-kube-api-access-84d5c" pod "9b0e06a0-bc8e-4cb5-b81c-b85942339178" (UID: "9b0e06a0-bc8e-4cb5-b81c-b85942339178"). InnerVolumeSpecName "kube-api-access-84d5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:34:48.710918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.710880    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84d5c\" (UniqueName: \"kubernetes.io/projected/9b0e06a0-bc8e-4cb5-b81c-b85942339178-kube-api-access-84d5c\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:34:48.914572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.914495    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6201e1a-34ee-4c94-b43a-9fc083e98810" path="/var/lib/kubelet/pods/a6201e1a-34ee-4c94-b43a-9fc083e98810/volumes"
Nov 02 21:34:48.914923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:48.914909    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4" path="/var/lib/kubelet/pods/c32a9afe-ecfe-4378-ad16-5a0ff32e9eb4/volumes"
Nov 02 21:34:49.260205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:49.260087    4528 generic.go:356] "Generic (PLEG): container finished" podID="9b0e06a0-bc8e-4cb5-b81c-b85942339178" containerID="e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926" exitCode=0
Nov 02 21:34:49.260205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:49.260147    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"
Nov 02 21:34:49.260205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:49.260168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj" event={"ID":"9b0e06a0-bc8e-4cb5-b81c-b85942339178","Type":"ContainerDied","Data":"e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926"}
Nov 02 21:34:49.260731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:49.260223    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj" event={"ID":"9b0e06a0-bc8e-4cb5-b81c-b85942339178","Type":"ContainerDied","Data":"b88a59d34a937f4b08b43438d3d314b475507ac427df1ec350735d226ea2b118"}
Nov 02 21:34:49.260731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:49.260246    4528 scope.go:122] "RemoveContainer" containerID="e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926"
Nov 02 21:34:49.275351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:49.275329    4528 scope.go:122] "RemoveContainer" containerID="e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926"
Nov 02 21:34:49.275682 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:34:49.275659    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926\": container with ID starting with e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926 not found: ID does not exist" containerID="e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926"
Nov 02 21:34:49.275748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:49.275690    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926"} err="failed to get container status \"e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926\": rpc error: code = NotFound desc = could not find container \"e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926\": container with ID starting with e5b49c2a41711b8b96e1dda99525cd3883a77e917a37f7e48cf5a7d44ceea926 not found: ID does not exist"
Nov 02 21:34:49.428227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:49.428193    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"]
Nov 02 21:34:49.478570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:49.478538    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d58d18-7b959d6655-w6wfj"]
Nov 02 21:34:50.914663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:50.914632    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0e06a0-bc8e-4cb5-b81c-b85942339178" path="/var/lib/kubelet/pods/9b0e06a0-bc8e-4cb5-b81c-b85942339178/volumes"
Nov 02 21:34:52.469730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:52.469681    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:34:53.053919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:53.053030    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"]
Nov 02 21:34:53.104709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:53.104652    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46d4b76-6pfhm"]
Nov 02 21:34:54.914678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:34:54.914646    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec79f71-7b45-4942-a5b2-a6453ce198cf" path="/var/lib/kubelet/pods/1ec79f71-7b45-4942-a5b2-a6453ce198cf/volumes"
Nov 02 21:35:02.050549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:02.050511    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"]
Nov 02 21:35:02.101830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:02.101792    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc785ce6-lnv7v"]
Nov 02 21:35:02.914754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:02.914719    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0c3f31-1d14-4535-bfc5-4af2e0233682" path="/var/lib/kubelet/pods/0b0c3f31-1d14-4535-bfc5-4af2e0233682/volumes"
Nov 02 21:35:14.114043 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:14.114005    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:35:14.976770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:14.976736    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"]
Nov 02 21:35:14.977042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:14.976974    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy" containerID="cri-o://fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800" gracePeriod=30
Nov 02 21:35:14.995855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:14.995820    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"]
Nov 02 21:35:14.996026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:14.996007    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy" containerID="cri-o://8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95" gracePeriod=30
Nov 02 21:35:15.027078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:15.027041    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"]
Nov 02 21:35:15.027331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:15.027300    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy" containerID="cri-o://e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa" gracePeriod=30
Nov 02 21:35:15.133761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:15.132330    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"]
Nov 02 21:35:15.144305 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:15.144166    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3298cc19-27swk"]
Nov 02 21:35:16.914121 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:16.914075    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0d93c9-4b6d-4e98-938a-75b595a149a9" path="/var/lib/kubelet/pods/9d0d93c9-4b6d-4e98-938a-75b595a149a9/volumes"
Nov 02 21:35:17.648918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:17.648879    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:35:17.649130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:17.648971    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:35:17.711582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:17.711550    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:35:17.711730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:17.711613    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:35:18.268105 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:18.268066    4528 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:35:18.268551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:18.268173    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:35:20.326689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.326655    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:35:20.377367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.377338    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.389726    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-podinfo\") pod \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.389785    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-podinfo\") pod \"52228fc8-539f-40bd-9f70-9c08cfed5739\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.389822    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-socket\") pod \"52228fc8-539f-40bd-9f70-9c08cfed5739\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.389876    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-socket\") pod \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.389955    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-kube-api-access-9ls9v\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-kube-api-access-9ls9v\") pod \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.389993    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-data\") pod \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390045    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istiod-ca-cert\") pod \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390087    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-envoy\") pod \"52228fc8-539f-40bd-9f70-9c08cfed5739\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390131    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52228fc8-539f-40bd-9f70-9c08cfed5739-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52228fc8-539f-40bd-9f70-9c08cfed5739-istiod-ca-cert\") pod \"52228fc8-539f-40bd-9f70-9c08cfed5739\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390219    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-certs\") pod \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390290    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-credential-socket\") pod \"52228fc8-539f-40bd-9f70-9c08cfed5739\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390372    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-envoy\") pod \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390408    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-certs\") pod \"52228fc8-539f-40bd-9f70-9c08cfed5739\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390448    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-token\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-token\") pod \"52228fc8-539f-40bd-9f70-9c08cfed5739\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390520    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-credential-socket\") pod \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390547    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-data" pod "6fbc8da9-f9e7-4b71-b25e-71beccf3d298" (UID: "6fbc8da9-f9e7-4b71-b25e-71beccf3d298"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390567    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-data\") pod \"52228fc8-539f-40bd-9f70-9c08cfed5739\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390657    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-token\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-token\") pod \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\" (UID: \"6fbc8da9-f9e7-4b71-b25e-71beccf3d298\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390707    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-kube-api-access-gdlbs\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-kube-api-access-gdlbs\") pod \"52228fc8-539f-40bd-9f70-9c08cfed5739\" (UID: \"52228fc8-539f-40bd-9f70-9c08cfed5739\") "
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.390717    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-socket" pod "52228fc8-539f-40bd-9f70-9c08cfed5739" (UID: "52228fc8-539f-40bd-9f70-9c08cfed5739"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.391129    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52228fc8-539f-40bd-9f70-9c08cfed5739-istiod-ca-cert" pod "52228fc8-539f-40bd-9f70-9c08cfed5739" (UID: "52228fc8-539f-40bd-9f70-9c08cfed5739"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.391400    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.391418    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.391426    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52228fc8-539f-40bd-9f70-9c08cfed5739-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.391707    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istiod-ca-cert" pod "6fbc8da9-f9e7-4b71-b25e-71beccf3d298" (UID: "6fbc8da9-f9e7-4b71-b25e-71beccf3d298"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.391916    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-socket" pod "6fbc8da9-f9e7-4b71-b25e-71beccf3d298" (UID: "6fbc8da9-f9e7-4b71-b25e-71beccf3d298"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.392441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.392407    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-credential-socket" pod "6fbc8da9-f9e7-4b71-b25e-71beccf3d298" (UID: "6fbc8da9-f9e7-4b71-b25e-71beccf3d298"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.393967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.392548    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-podinfo" pod "52228fc8-539f-40bd-9f70-9c08cfed5739" (UID: "52228fc8-539f-40bd-9f70-9c08cfed5739"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:35:20.393967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.392580    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-certs" pod "52228fc8-539f-40bd-9f70-9c08cfed5739" (UID: "52228fc8-539f-40bd-9f70-9c08cfed5739"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.393967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.392761    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-data" pod "52228fc8-539f-40bd-9f70-9c08cfed5739" (UID: "52228fc8-539f-40bd-9f70-9c08cfed5739"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.393967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.392967    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-certs" pod "6fbc8da9-f9e7-4b71-b25e-71beccf3d298" (UID: "6fbc8da9-f9e7-4b71-b25e-71beccf3d298"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.393967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.393190    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-credential-socket" pod "52228fc8-539f-40bd-9f70-9c08cfed5739" (UID: "52228fc8-539f-40bd-9f70-9c08cfed5739"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.393967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.393860    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-kube-api-access-gdlbs" pod "52228fc8-539f-40bd-9f70-9c08cfed5739" (UID: "52228fc8-539f-40bd-9f70-9c08cfed5739"). InnerVolumeSpecName "kube-api-access-gdlbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:20.394579 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.394556    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-token" pod "6fbc8da9-f9e7-4b71-b25e-71beccf3d298" (UID: "6fbc8da9-f9e7-4b71-b25e-71beccf3d298"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:20.395033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.395009    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-token" pod "52228fc8-539f-40bd-9f70-9c08cfed5739" (UID: "52228fc8-539f-40bd-9f70-9c08cfed5739"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:20.395461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.395435    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-kube-api-access-9ls9v" pod "6fbc8da9-f9e7-4b71-b25e-71beccf3d298" (UID: "6fbc8da9-f9e7-4b71-b25e-71beccf3d298"). InnerVolumeSpecName "kube-api-access-9ls9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:20.395969 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.395941    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-envoy" pod "6fbc8da9-f9e7-4b71-b25e-71beccf3d298" (UID: "6fbc8da9-f9e7-4b71-b25e-71beccf3d298"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.396045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.395974    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-podinfo" pod "6fbc8da9-f9e7-4b71-b25e-71beccf3d298" (UID: "6fbc8da9-f9e7-4b71-b25e-71beccf3d298"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:35:20.396551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.396532    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-envoy" pod "52228fc8-539f-40bd-9f70-9c08cfed5739" (UID: "52228fc8-539f-40bd-9f70-9c08cfed5739"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.399189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.399098    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:35:20.492903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.492866    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-envoy\") pod \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") "
Nov 02 21:35:20.493144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.492933    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-certs\") pod \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") "
Nov 02 21:35:20.493144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493007    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istiod-ca-cert\") pod \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") "
Nov 02 21:35:20.493144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493089    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-data\") pod \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") "
Nov 02 21:35:20.493144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493137    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-credential-socket\") pod \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") "
Nov 02 21:35:20.493396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493206    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-token\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-token\") pod \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") "
Nov 02 21:35:20.493396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493243    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-podinfo\") pod \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") "
Nov 02 21:35:20.493396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493270    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-socket\") pod \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") "
Nov 02 21:35:20.493396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493334    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-certs" pod "9c9df425-9d79-4cf9-8c80-38b4a114aba6" (UID: "9c9df425-9d79-4cf9-8c80-38b4a114aba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.493396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493365    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-kube-api-access-426t9\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-kube-api-access-426t9\") pod \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\" (UID: \"9c9df425-9d79-4cf9-8c80-38b4a114aba6\") "
Nov 02 21:35:20.493652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493473    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-credential-socket" pod "9c9df425-9d79-4cf9-8c80-38b4a114aba6" (UID: "9c9df425-9d79-4cf9-8c80-38b4a114aba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.493652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493493    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istiod-ca-cert" pod "9c9df425-9d79-4cf9-8c80-38b4a114aba6" (UID: "9c9df425-9d79-4cf9-8c80-38b4a114aba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:35:20.493994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493765    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-socket" pod "9c9df425-9d79-4cf9-8c80-38b4a114aba6" (UID: "9c9df425-9d79-4cf9-8c80-38b4a114aba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.493994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.493930    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-data" pod "9c9df425-9d79-4cf9-8c80-38b4a114aba6" (UID: "9c9df425-9d79-4cf9-8c80-38b4a114aba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.494266 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494242    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdlbs\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-kube-api-access-gdlbs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494274    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494290    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494305    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494320    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494334    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494348    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494362    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494376    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ls9v\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-kube-api-access-9ls9v\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494389    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494403    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494419    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494433    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494447    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494460    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494474    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494489    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494502    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494517    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52228fc8-539f-40bd-9f70-9c08cfed5739-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.494589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.494530    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fbc8da9-f9e7-4b71-b25e-71beccf3d298-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.495674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.495649    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-envoy" pod "9c9df425-9d79-4cf9-8c80-38b4a114aba6" (UID: "9c9df425-9d79-4cf9-8c80-38b4a114aba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:20.495776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.495649    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-kube-api-access-426t9" pod "9c9df425-9d79-4cf9-8c80-38b4a114aba6" (UID: "9c9df425-9d79-4cf9-8c80-38b4a114aba6"). InnerVolumeSpecName "kube-api-access-426t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:20.496093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.496064    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-podinfo" pod "9c9df425-9d79-4cf9-8c80-38b4a114aba6" (UID: "9c9df425-9d79-4cf9-8c80-38b4a114aba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:35:20.496266 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.496239    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-token" pod "9c9df425-9d79-4cf9-8c80-38b4a114aba6" (UID: "9c9df425-9d79-4cf9-8c80-38b4a114aba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:20.596063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.596023    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.596063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.596060    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.596288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.596077    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-426t9\" (UniqueName: \"kubernetes.io/projected/9c9df425-9d79-4cf9-8c80-38b4a114aba6-kube-api-access-426t9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.596288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.596092    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9df425-9d79-4cf9-8c80-38b4a114aba6-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:20.638132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.638037    4528 generic.go:356] "Generic (PLEG): container finished" podID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerID="fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800" exitCode=0
Nov 02 21:35:20.638328 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.638283    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" event={"ID":"6fbc8da9-f9e7-4b71-b25e-71beccf3d298","Type":"ContainerDied","Data":"fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800"}
Nov 02 21:35:20.638401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.638327    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"
Nov 02 21:35:20.638459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.638395    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt" event={"ID":"6fbc8da9-f9e7-4b71-b25e-71beccf3d298","Type":"ContainerDied","Data":"2fe5bffd680c7ee6a6ee1abcd6a081d007559219a603df2cdd4db8a41a5e286d"}
Nov 02 21:35:20.654165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.638437    4528 scope.go:122] "RemoveContainer" containerID="fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800"
Nov 02 21:35:20.654165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.640509    4528 generic.go:356] "Generic (PLEG): container finished" podID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerID="e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa" exitCode=0
Nov 02 21:35:20.654165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.640630    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" event={"ID":"9c9df425-9d79-4cf9-8c80-38b4a114aba6","Type":"ContainerDied","Data":"e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa"}
Nov 02 21:35:20.654165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.640678    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt" event={"ID":"9c9df425-9d79-4cf9-8c80-38b4a114aba6","Type":"ContainerDied","Data":"e9b26c5ce22d603e03c5734313a8d5c916c8899ac7f7e03a06971ae3f59dcc36"}
Nov 02 21:35:20.654165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.640709    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"
Nov 02 21:35:20.654165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.642797    4528 generic.go:356] "Generic (PLEG): container finished" podID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerID="8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95" exitCode=0
Nov 02 21:35:20.654165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.642914    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" event={"ID":"52228fc8-539f-40bd-9f70-9c08cfed5739","Type":"ContainerDied","Data":"8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95"}
Nov 02 21:35:20.654165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.642961    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb" event={"ID":"52228fc8-539f-40bd-9f70-9c08cfed5739","Type":"ContainerDied","Data":"d7c7234d879e0974d5443520e701f3e2d06a6c49d7f330071622445ad1347702"}
Nov 02 21:35:20.654165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.643072    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"
Nov 02 21:35:20.661227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.661203    4528 scope.go:122] "RemoveContainer" containerID="fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800"
Nov 02 21:35:20.661578 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:20.661553    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800\": container with ID starting with fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800 not found: ID does not exist" containerID="fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800"
Nov 02 21:35:20.661657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.661588    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800"} err="failed to get container status \"fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800\": rpc error: code = NotFound desc = could not find container \"fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800\": container with ID starting with fe0c1f1aff0b594c8fc17c4917dc0ddd8a7289d1ceb2217037688230e0c52800 not found: ID does not exist"
Nov 02 21:35:20.661657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.661607    4528 scope.go:122] "RemoveContainer" containerID="e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa"
Nov 02 21:35:20.679693 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.679672    4528 scope.go:122] "RemoveContainer" containerID="e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa"
Nov 02 21:35:20.680036 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:20.680015    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa\": container with ID starting with e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa not found: ID does not exist" containerID="e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa"
Nov 02 21:35:20.680148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.680046    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa"} err="failed to get container status \"e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa\": rpc error: code = NotFound desc = could not find container \"e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa\": container with ID starting with e74b2f266aaa56d393bf457e5f0fb6e7362caf90a55df5c57e4db10a8b867efa not found: ID does not exist"
Nov 02 21:35:20.680148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.680067    4528 scope.go:122] "RemoveContainer" containerID="8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95"
Nov 02 21:35:20.690036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.690017    4528 scope.go:122] "RemoveContainer" containerID="8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95"
Nov 02 21:35:20.690348 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:20.690330    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95\": container with ID starting with 8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95 not found: ID does not exist" containerID="8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95"
Nov 02 21:35:20.690422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.690354    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95"} err="failed to get container status \"8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95\": rpc error: code = NotFound desc = could not find container \"8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95\": container with ID starting with 8621297c01494a001f1926e490ff793dd0c0b4ff8aa71fd7e336061952ff9e95 not found: ID does not exist"
Nov 02 21:35:20.848146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.848099    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"]
Nov 02 21:35:20.852303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.852275    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-27qlt"]
Nov 02 21:35:20.915892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.915862    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" path="/var/lib/kubelet/pods/9c9df425-9d79-4cf9-8c80-38b4a114aba6/volumes"
Nov 02 21:35:20.961422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.961387    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"]
Nov 02 21:35:20.965435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:20.965409    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-5d4mb"]
Nov 02 21:35:21.075486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:21.075453    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"]
Nov 02 21:35:21.078913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:21.078892    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-ffmnt"]
Nov 02 21:35:22.845319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:22.845276    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"]
Nov 02 21:35:22.914997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:22.914965    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" path="/var/lib/kubelet/pods/52228fc8-539f-40bd-9f70-9c08cfed5739/volumes"
Nov 02 21:35:22.915555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:22.915539    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" path="/var/lib/kubelet/pods/6fbc8da9-f9e7-4b71-b25e-71beccf3d298/volumes"
Nov 02 21:35:22.964971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:22.964922    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"]
Nov 02 21:35:25.828517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:25.828478    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"]
Nov 02 21:35:25.829037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:25.828747    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy" containerID="cri-o://5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33" gracePeriod=30
Nov 02 21:35:27.846349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:27.846295    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk" podUID="21279019-ae8a-42a2-b6c0-691809a777ea" containerName="payload-processing" containerID="cri-o://114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0" gracePeriod=25
Nov 02 21:35:27.966504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:27.966469    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj" podUID="970f66e4-08da-4bff-8533-1e81a9fddbb6" containerName="payload-pre-processing" containerID="cri-o://5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa" gracePeriod=25
Nov 02 21:35:28.146274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.146242    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:35:28.241411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.241381    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21279019-ae8a-42a2-b6c0-691809a777ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21279019-ae8a-42a2-b6c0-691809a777ea-plugins-config-volume\") pod \"21279019-ae8a-42a2-b6c0-691809a777ea\" (UID: \"21279019-ae8a-42a2-b6c0-691809a777ea\") "
Nov 02 21:35:28.241588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.241576    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21279019-ae8a-42a2-b6c0-691809a777ea-kube-api-access-ld5f7\" (UniqueName: \"kubernetes.io/projected/21279019-ae8a-42a2-b6c0-691809a777ea-kube-api-access-ld5f7\") pod \"21279019-ae8a-42a2-b6c0-691809a777ea\" (UID: \"21279019-ae8a-42a2-b6c0-691809a777ea\") "
Nov 02 21:35:28.241766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.241740    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21279019-ae8a-42a2-b6c0-691809a777ea-plugins-config-volume" pod "21279019-ae8a-42a2-b6c0-691809a777ea" (UID: "21279019-ae8a-42a2-b6c0-691809a777ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:35:28.242056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.242041    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21279019-ae8a-42a2-b6c0-691809a777ea-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:28.243703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.243682    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21279019-ae8a-42a2-b6c0-691809a777ea-kube-api-access-ld5f7" pod "21279019-ae8a-42a2-b6c0-691809a777ea" (UID: "21279019-ae8a-42a2-b6c0-691809a777ea"). InnerVolumeSpecName "kube-api-access-ld5f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:28.263758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.263736    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:35:28.342977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.342945    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970f66e4-08da-4bff-8533-1e81a9fddbb6-kube-api-access-699fq\" (UniqueName: \"kubernetes.io/projected/970f66e4-08da-4bff-8533-1e81a9fddbb6-kube-api-access-699fq\") pod \"970f66e4-08da-4bff-8533-1e81a9fddbb6\" (UID: \"970f66e4-08da-4bff-8533-1e81a9fddbb6\") "
Nov 02 21:35:28.343211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.343092    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/970f66e4-08da-4bff-8533-1e81a9fddbb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970f66e4-08da-4bff-8533-1e81a9fddbb6-plugins-config-volume\") pod \"970f66e4-08da-4bff-8533-1e81a9fddbb6\" (UID: \"970f66e4-08da-4bff-8533-1e81a9fddbb6\") "
Nov 02 21:35:28.343463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.343440    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970f66e4-08da-4bff-8533-1e81a9fddbb6-plugins-config-volume" pod "970f66e4-08da-4bff-8533-1e81a9fddbb6" (UID: "970f66e4-08da-4bff-8533-1e81a9fddbb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:35:28.343551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.343538    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5f7\" (UniqueName: \"kubernetes.io/projected/21279019-ae8a-42a2-b6c0-691809a777ea-kube-api-access-ld5f7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:28.343597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.343552    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970f66e4-08da-4bff-8533-1e81a9fddbb6-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:28.345025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.345005    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970f66e4-08da-4bff-8533-1e81a9fddbb6-kube-api-access-699fq" pod "970f66e4-08da-4bff-8533-1e81a9fddbb6" (UID: "970f66e4-08da-4bff-8533-1e81a9fddbb6"). InnerVolumeSpecName "kube-api-access-699fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:28.445315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.445230    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-699fq\" (UniqueName: \"kubernetes.io/projected/970f66e4-08da-4bff-8533-1e81a9fddbb6-kube-api-access-699fq\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:28.793420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.793380    4528 generic.go:356] "Generic (PLEG): container finished" podID="21279019-ae8a-42a2-b6c0-691809a777ea" containerID="114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0" exitCode=0
Nov 02 21:35:28.793610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.793442    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"
Nov 02 21:35:28.793610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.793454    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk" event={"ID":"21279019-ae8a-42a2-b6c0-691809a777ea","Type":"ContainerDied","Data":"114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0"}
Nov 02 21:35:28.793610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.793529    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk" event={"ID":"21279019-ae8a-42a2-b6c0-691809a777ea","Type":"ContainerDied","Data":"6562eca3b5f43f70258c012f19ddcd95fd8db69bcf1a4188ee168632534d962d"}
Nov 02 21:35:28.793610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.793551    4528 scope.go:122] "RemoveContainer" containerID="114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0"
Nov 02 21:35:28.795311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.795285    4528 generic.go:356] "Generic (PLEG): container finished" podID="970f66e4-08da-4bff-8533-1e81a9fddbb6" containerID="5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa" exitCode=0
Nov 02 21:35:28.795439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.795325    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj" event={"ID":"970f66e4-08da-4bff-8533-1e81a9fddbb6","Type":"ContainerDied","Data":"5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa"}
Nov 02 21:35:28.795439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.795353    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"
Nov 02 21:35:28.795439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.795364    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj" event={"ID":"970f66e4-08da-4bff-8533-1e81a9fddbb6","Type":"ContainerDied","Data":"36496584eefe71d9a91429d941a5c3ca008183d10a56cf2c8f1ef8604ea1e520"}
Nov 02 21:35:28.806264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.806242    4528 scope.go:122] "RemoveContainer" containerID="114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0"
Nov 02 21:35:28.806540 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:28.806522    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0\": container with ID starting with 114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0 not found: ID does not exist" containerID="114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0"
Nov 02 21:35:28.806596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.806550    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0"} err="failed to get container status \"114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0\": rpc error: code = NotFound desc = could not find container \"114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0\": container with ID starting with 114760128b552447e1568f742d0221e1c858c5d9265265e83bc3349c5f1183e0 not found: ID does not exist"
Nov 02 21:35:28.806596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.806567    4528 scope.go:122] "RemoveContainer" containerID="5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa"
Nov 02 21:35:28.815609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.815580    4528 scope.go:122] "RemoveContainer" containerID="5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa"
Nov 02 21:35:28.815922 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:28.815900    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa\": container with ID starting with 5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa not found: ID does not exist" containerID="5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa"
Nov 02 21:35:28.815998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:28.815932    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa"} err="failed to get container status \"5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa\": rpc error: code = NotFound desc = could not find container \"5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa\": container with ID starting with 5b34aaa25f655e32c530afaf37cd45f4e795bc2dab210fd31a98d8d6391935fa not found: ID does not exist"
Nov 02 21:35:29.025415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:29.025378    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"]
Nov 02 21:35:29.108320 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:29.106915    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54a4b2-68d494c969-994gk"]
Nov 02 21:35:29.175132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:29.173931    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"]
Nov 02 21:35:29.187213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:29.181158    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-54a4b2-7f4df7975f-6vmsj"]
Nov 02 21:35:30.933910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:30.933880    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21279019-ae8a-42a2-b6c0-691809a777ea" path="/var/lib/kubelet/pods/21279019-ae8a-42a2-b6c0-691809a777ea/volumes"
Nov 02 21:35:30.934389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:30.934373    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970f66e4-08da-4bff-8533-1e81a9fddbb6" path="/var/lib/kubelet/pods/970f66e4-08da-4bff-8533-1e81a9fddbb6/volumes"
Nov 02 21:35:31.103808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.103745    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:35:31.210448 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.210415    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-token\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-token\") pod \"47468c66-6cd7-4d1e-a705-0689ed95559a\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") "
Nov 02 21:35:31.211283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.211259    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-certs\") pod \"47468c66-6cd7-4d1e-a705-0689ed95559a\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") "
Nov 02 21:35:31.212035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.212015    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-envoy\") pod \"47468c66-6cd7-4d1e-a705-0689ed95559a\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") "
Nov 02 21:35:31.213024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.213006    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-podinfo\") pod \"47468c66-6cd7-4d1e-a705-0689ed95559a\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") "
Nov 02 21:35:31.213575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.213547    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-kube-api-access-7l2l9\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-kube-api-access-7l2l9\") pod \"47468c66-6cd7-4d1e-a705-0689ed95559a\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") "
Nov 02 21:35:31.213779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.213756    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-credential-socket\") pod \"47468c66-6cd7-4d1e-a705-0689ed95559a\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") "
Nov 02 21:35:31.213981 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.213966    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-socket\") pod \"47468c66-6cd7-4d1e-a705-0689ed95559a\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") "
Nov 02 21:35:31.214204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.214169    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47468c66-6cd7-4d1e-a705-0689ed95559a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47468c66-6cd7-4d1e-a705-0689ed95559a-istiod-ca-cert\") pod \"47468c66-6cd7-4d1e-a705-0689ed95559a\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") "
Nov 02 21:35:31.214417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.214389    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-data\") pod \"47468c66-6cd7-4d1e-a705-0689ed95559a\" (UID: \"47468c66-6cd7-4d1e-a705-0689ed95559a\") "
Nov 02 21:35:31.215445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.211843    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-certs" pod "47468c66-6cd7-4d1e-a705-0689ed95559a" (UID: "47468c66-6cd7-4d1e-a705-0689ed95559a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:31.215445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.214610    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-socket" pod "47468c66-6cd7-4d1e-a705-0689ed95559a" (UID: "47468c66-6cd7-4d1e-a705-0689ed95559a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:31.215734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.214980    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-credential-socket" pod "47468c66-6cd7-4d1e-a705-0689ed95559a" (UID: "47468c66-6cd7-4d1e-a705-0689ed95559a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:31.215734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.215394    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47468c66-6cd7-4d1e-a705-0689ed95559a-istiod-ca-cert" pod "47468c66-6cd7-4d1e-a705-0689ed95559a" (UID: "47468c66-6cd7-4d1e-a705-0689ed95559a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:35:31.215980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.215961    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-data" pod "47468c66-6cd7-4d1e-a705-0689ed95559a" (UID: "47468c66-6cd7-4d1e-a705-0689ed95559a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:31.234104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.227743    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-envoy" pod "47468c66-6cd7-4d1e-a705-0689ed95559a" (UID: "47468c66-6cd7-4d1e-a705-0689ed95559a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:31.234104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.229376    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-token" pod "47468c66-6cd7-4d1e-a705-0689ed95559a" (UID: "47468c66-6cd7-4d1e-a705-0689ed95559a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:31.234104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.233693    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-podinfo" pod "47468c66-6cd7-4d1e-a705-0689ed95559a" (UID: "47468c66-6cd7-4d1e-a705-0689ed95559a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:35:31.235722 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.235678    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-kube-api-access-7l2l9" pod "47468c66-6cd7-4d1e-a705-0689ed95559a" (UID: "47468c66-6cd7-4d1e-a705-0689ed95559a"). InnerVolumeSpecName "kube-api-access-7l2l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:31.316819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.316784    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47468c66-6cd7-4d1e-a705-0689ed95559a-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:31.316819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.316813    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:31.317060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.316829    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:31.317060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.316841    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:31.317060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.316852    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:31.317060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.316864    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47468c66-6cd7-4d1e-a705-0689ed95559a-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:31.317060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.316875    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l2l9\" (UniqueName: \"kubernetes.io/projected/47468c66-6cd7-4d1e-a705-0689ed95559a-kube-api-access-7l2l9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:31.317060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.316889    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:31.317060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.316901    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47468c66-6cd7-4d1e-a705-0689ed95559a-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:31.820041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.820010    4528 generic.go:356] "Generic (PLEG): container finished" podID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerID="5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33" exitCode=0
Nov 02 21:35:31.820250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.820056    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"
Nov 02 21:35:31.820250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.820064    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" event={"ID":"47468c66-6cd7-4d1e-a705-0689ed95559a","Type":"ContainerDied","Data":"5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33"}
Nov 02 21:35:31.820250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.820104    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7" event={"ID":"47468c66-6cd7-4d1e-a705-0689ed95559a","Type":"ContainerDied","Data":"3733a69e91dcea54fba6b95acbf7430687c3103cefadc5b8f0ce56a5423b3b15"}
Nov 02 21:35:31.820250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.820141    4528 scope.go:122] "RemoveContainer" containerID="5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33"
Nov 02 21:35:31.831209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.831191    4528 scope.go:122] "RemoveContainer" containerID="5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33"
Nov 02 21:35:31.831457 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:31.831440    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33\": container with ID starting with 5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33 not found: ID does not exist" containerID="5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33"
Nov 02 21:35:31.831526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:31.831465    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33"} err="failed to get container status \"5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33\": rpc error: code = NotFound desc = could not find container \"5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33\": container with ID starting with 5d0944b9524310caeac92d0eaf2be749556273b0440db44d8c391b03a44b4b33 not found: ID does not exist"
Nov 02 21:35:32.012785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:32.012754    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"]
Nov 02 21:35:32.017227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:32.017192    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-44fz7"]
Nov 02 21:35:32.817707 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:32.817672    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"]
Nov 02 21:35:32.817968 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:32.817944    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy" containerID="cri-o://3f5fed2b3bb65484fd6c0f0468dbca971f2f9a02758a1765b0439abcdf1e7d07" gracePeriod=30
Nov 02 21:35:33.151881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:33.151784    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" path="/var/lib/kubelet/pods/47468c66-6cd7-4d1e-a705-0689ed95559a/volumes"
Nov 02 21:35:33.575350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:33.575318    4528 scope.go:122] "RemoveContainer" containerID="f795729ffabc6622e9494435fd408718d41781c3a1ab8bf5c5d3236430a69350"
Nov 02 21:35:33.585161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:33.585140    4528 scope.go:122] "RemoveContainer" containerID="a9fa6fcb170fc12d224090f51d8c603debdcd012a1f1924473e55138aac6dfa4"
Nov 02 21:35:33.594041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:33.594024    4528 scope.go:122] "RemoveContainer" containerID="ec1b5342772bb18d436d546bee61b698281b20cc9c16b119956abcc1f03f2e69"
Nov 02 21:35:33.624383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:33.605757    4528 scope.go:122] "RemoveContainer" containerID="fa1c38134473c93144133cddf9e6e9f026c677bcc028ce2808808f918a65052e"
Nov 02 21:35:33.670285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:33.670235    4528 scope.go:122] "RemoveContainer" containerID="4689780761d377e86d4d857dbe93c539c0bd2f3f94d6e15d8f9dbf6bbcedbab9"
Nov 02 21:35:33.679182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:33.679162    4528 scope.go:122] "RemoveContainer" containerID="4928d24e4b1b57bfbc3500fd3cf889f1614109069ee004c1af68dbf14ff1d3e4"
Nov 02 21:35:34.047982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:34.047946    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"]
Nov 02 21:35:34.097275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:34.097244    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6393175-zh5bz"]
Nov 02 21:35:34.987158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:34.986382    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24" path="/var/lib/kubelet/pods/a9611da1-8cb5-4d6e-a45a-b1e75c2b5e24/volumes"
Nov 02 21:35:36.061825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:36.059965    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"]
Nov 02 21:35:36.103078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:36.103048    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190dba28-mdgs2"]
Nov 02 21:35:36.922998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:36.922968    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6025d5-7b33-44ed-93d4-4ce861ae88e8" path="/var/lib/kubelet/pods/2a6025d5-7b33-44ed-93d4-4ce861ae88e8/volumes"
Nov 02 21:35:38.012619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.012587    4528 generic.go:356] "Generic (PLEG): container finished" podID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerID="3f5fed2b3bb65484fd6c0f0468dbca971f2f9a02758a1765b0439abcdf1e7d07" exitCode=0
Nov 02 21:35:38.013176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.012656    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" event={"ID":"36ed76ea-1ce8-4b37-b55c-5c98e5eef187","Type":"ContainerDied","Data":"3f5fed2b3bb65484fd6c0f0468dbca971f2f9a02758a1765b0439abcdf1e7d07"}
Nov 02 21:35:38.013176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.012724    4528 scope.go:122] "RemoveContainer" containerID="2c598e39c01a75d199878f0c10aeb61eda88c8ffba6d1aa8c39cf3ab13e4cae4"
Nov 02 21:35:38.091606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.091467    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"]
Nov 02 21:35:38.139033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.138998    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-cc79s"]
Nov 02 21:35:38.178252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.178230    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:35:38.287039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287009    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-certs\") pod \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") "
Nov 02 21:35:38.287247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287066    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-kube-api-access-vknx4\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-kube-api-access-vknx4\") pod \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") "
Nov 02 21:35:38.287247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287095    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-token\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-token\") pod \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") "
Nov 02 21:35:38.287247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287142    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-socket\") pod \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") "
Nov 02 21:35:38.287247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287199    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-data\") pod \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") "
Nov 02 21:35:38.287247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287249    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istiod-ca-cert\") pod \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") "
Nov 02 21:35:38.287513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287270    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-credential-socket\") pod \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") "
Nov 02 21:35:38.287513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287275    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-certs" pod "36ed76ea-1ce8-4b37-b55c-5c98e5eef187" (UID: "36ed76ea-1ce8-4b37-b55c-5c98e5eef187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:38.287513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287403    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-envoy\") pod \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") "
Nov 02 21:35:38.287513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287448    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-socket" pod "36ed76ea-1ce8-4b37-b55c-5c98e5eef187" (UID: "36ed76ea-1ce8-4b37-b55c-5c98e5eef187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:38.287513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287504    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-podinfo\") pod \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\" (UID: \"36ed76ea-1ce8-4b37-b55c-5c98e5eef187\") "
Nov 02 21:35:38.287736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287525    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-data" pod "36ed76ea-1ce8-4b37-b55c-5c98e5eef187" (UID: "36ed76ea-1ce8-4b37-b55c-5c98e5eef187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:38.287736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287538    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-credential-socket" pod "36ed76ea-1ce8-4b37-b55c-5c98e5eef187" (UID: "36ed76ea-1ce8-4b37-b55c-5c98e5eef187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:38.287736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.287640    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istiod-ca-cert" pod "36ed76ea-1ce8-4b37-b55c-5c98e5eef187" (UID: "36ed76ea-1ce8-4b37-b55c-5c98e5eef187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:35:38.288213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.288192    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:38.288213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.288211    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:38.288392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.288221    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:38.288392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.288230    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:38.288392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.288238    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:38.289573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.289547    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-token" pod "36ed76ea-1ce8-4b37-b55c-5c98e5eef187" (UID: "36ed76ea-1ce8-4b37-b55c-5c98e5eef187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:38.289655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.289607    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-podinfo" pod "36ed76ea-1ce8-4b37-b55c-5c98e5eef187" (UID: "36ed76ea-1ce8-4b37-b55c-5c98e5eef187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:35:38.289795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.289778    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-kube-api-access-vknx4" pod "36ed76ea-1ce8-4b37-b55c-5c98e5eef187" (UID: "36ed76ea-1ce8-4b37-b55c-5c98e5eef187"). InnerVolumeSpecName "kube-api-access-vknx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:38.289858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.289840    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-envoy" pod "36ed76ea-1ce8-4b37-b55c-5c98e5eef187" (UID: "36ed76ea-1ce8-4b37-b55c-5c98e5eef187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:38.390100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.390021    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vknx4\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-kube-api-access-vknx4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:38.390100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.390049    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:38.390100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.390059    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:38.390100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.390068    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ed76ea-1ce8-4b37-b55c-5c98e5eef187-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:38.913408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:38.913374    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130fde42-055a-4e04-b040-6b0931caa0e3" path="/var/lib/kubelet/pods/130fde42-055a-4e04-b040-6b0931caa0e3/volumes"
Nov 02 21:35:39.020379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:39.020348    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"
Nov 02 21:35:39.020379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:39.020367    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp" event={"ID":"36ed76ea-1ce8-4b37-b55c-5c98e5eef187","Type":"ContainerDied","Data":"0759927c54a177be1d8a6059bb394010de0d8b9221dfc76f8348c0cb38032338"}
Nov 02 21:35:39.020910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:39.020422    4528 scope.go:122] "RemoveContainer" containerID="3f5fed2b3bb65484fd6c0f0468dbca971f2f9a02758a1765b0439abcdf1e7d07"
Nov 02 21:35:39.238472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:39.238397    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"]
Nov 02 21:35:39.288286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:39.288257    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-03ece2-openshift-default-56d8d98b45-vzvjp"]
Nov 02 21:35:40.604598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:40.604564    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:35:40.968894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:40.964740    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" path="/var/lib/kubelet/pods/36ed76ea-1ce8-4b37-b55c-5c98e5eef187/volumes"
Nov 02 21:35:41.107021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:41.106986    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"]
Nov 02 21:35:44.114834 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:44.114796    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:35:44.276516 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:44.276480    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"]
Nov 02 21:35:44.276775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:44.276728    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy" containerID="cri-o://6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637" gracePeriod=30
Nov 02 21:35:45.077965 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:45.077933    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"]
Nov 02 21:35:45.110090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:45.110053    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0396ef3d-gbrhz"]
Nov 02 21:35:46.108280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:46.108233    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx" podUID="04442c1e-13fb-4c6d-af78-72e266feed42" containerName="payload-processing" containerID="cri-o://8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b" gracePeriod=25
Nov 02 21:35:46.412056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:46.412032    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:35:46.446764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:46.446738    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04442c1e-13fb-4c6d-af78-72e266feed42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04442c1e-13fb-4c6d-af78-72e266feed42-plugins-config-volume\") pod \"04442c1e-13fb-4c6d-af78-72e266feed42\" (UID: \"04442c1e-13fb-4c6d-af78-72e266feed42\") "
Nov 02 21:35:46.446947 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:46.446936    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04442c1e-13fb-4c6d-af78-72e266feed42-kube-api-access-r6ffs\" (UniqueName: \"kubernetes.io/projected/04442c1e-13fb-4c6d-af78-72e266feed42-kube-api-access-r6ffs\") pod \"04442c1e-13fb-4c6d-af78-72e266feed42\" (UID: \"04442c1e-13fb-4c6d-af78-72e266feed42\") "
Nov 02 21:35:46.447088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:46.447066    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04442c1e-13fb-4c6d-af78-72e266feed42-plugins-config-volume" pod "04442c1e-13fb-4c6d-af78-72e266feed42" (UID: "04442c1e-13fb-4c6d-af78-72e266feed42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:35:46.447333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:46.447318    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04442c1e-13fb-4c6d-af78-72e266feed42-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:46.448971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:46.448931    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04442c1e-13fb-4c6d-af78-72e266feed42-kube-api-access-r6ffs" pod "04442c1e-13fb-4c6d-af78-72e266feed42" (UID: "04442c1e-13fb-4c6d-af78-72e266feed42"). InnerVolumeSpecName "kube-api-access-r6ffs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:46.550991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:46.550452    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6ffs\" (UniqueName: \"kubernetes.io/projected/04442c1e-13fb-4c6d-af78-72e266feed42-kube-api-access-r6ffs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:46.913737 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:46.913692    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de84cbf-26d8-48a2-a432-260791fc5c74" path="/var/lib/kubelet/pods/9de84cbf-26d8-48a2-a432-260791fc5c74/volumes"
Nov 02 21:35:47.209289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:47.209199    4528 generic.go:356] "Generic (PLEG): container finished" podID="04442c1e-13fb-4c6d-af78-72e266feed42" containerID="8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b" exitCode=0
Nov 02 21:35:47.209289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:47.209241    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"
Nov 02 21:35:47.209289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:47.209255    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx" event={"ID":"04442c1e-13fb-4c6d-af78-72e266feed42","Type":"ContainerDied","Data":"8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b"}
Nov 02 21:35:47.209854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:47.209317    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx" event={"ID":"04442c1e-13fb-4c6d-af78-72e266feed42","Type":"ContainerDied","Data":"16d058298d59472d051705309ce4fe1b63384aaaf1b27b8a7d7bf8355e95f76f"}
Nov 02 21:35:47.209854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:47.209346    4528 scope.go:122] "RemoveContainer" containerID="8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b"
Nov 02 21:35:47.219706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:47.219687    4528 scope.go:122] "RemoveContainer" containerID="8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b"
Nov 02 21:35:47.220012 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:47.219993    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b\": container with ID starting with 8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b not found: ID does not exist" containerID="8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b"
Nov 02 21:35:47.220062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:47.220020    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b"} err="failed to get container status \"8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b\": rpc error: code = NotFound desc = could not find container \"8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b\": container with ID starting with 8b34786fe7eda8596ec7f8b0430d2a17c6a3e15fc489e5b1eafbeccfc94ddb6b not found: ID does not exist"
Nov 02 21:35:47.375483 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:47.375455    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"]
Nov 02 21:35:47.425909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:47.425870    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bd96ef-6d4f6db85c-4zqbx"]
Nov 02 21:35:48.913584 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:48.913550    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04442c1e-13fb-4c6d-af78-72e266feed42" path="/var/lib/kubelet/pods/04442c1e-13fb-4c6d-af78-72e266feed42/volumes"
Nov 02 21:35:49.561281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.561256    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:35:49.599743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.599719    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-socket\") pod \"1698d6b6-be71-4474-8747-2fc8a484770a\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") "
Nov 02 21:35:49.599881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.599803    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-kube-api-access-7fnmb\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-kube-api-access-7fnmb\") pod \"1698d6b6-be71-4474-8747-2fc8a484770a\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") "
Nov 02 21:35:49.599881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.599838    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-data\") pod \"1698d6b6-be71-4474-8747-2fc8a484770a\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") "
Nov 02 21:35:49.599949 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.599920    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1698d6b6-be71-4474-8747-2fc8a484770a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1698d6b6-be71-4474-8747-2fc8a484770a-istio-podinfo\") pod \"1698d6b6-be71-4474-8747-2fc8a484770a\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") "
Nov 02 21:35:49.600283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.599979    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1698d6b6-be71-4474-8747-2fc8a484770a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1698d6b6-be71-4474-8747-2fc8a484770a-istiod-ca-cert\") pod \"1698d6b6-be71-4474-8747-2fc8a484770a\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") "
Nov 02 21:35:49.600283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.600000    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-socket" pod "1698d6b6-be71-4474-8747-2fc8a484770a" (UID: "1698d6b6-be71-4474-8747-2fc8a484770a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:49.600283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.600037    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-istio-token\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-istio-token\") pod \"1698d6b6-be71-4474-8747-2fc8a484770a\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") "
Nov 02 21:35:49.600283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.600066    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-certs\") pod \"1698d6b6-be71-4474-8747-2fc8a484770a\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") "
Nov 02 21:35:49.600283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.600153    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-credential-socket\") pod \"1698d6b6-be71-4474-8747-2fc8a484770a\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") "
Nov 02 21:35:49.600283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.600193    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-envoy\") pod \"1698d6b6-be71-4474-8747-2fc8a484770a\" (UID: \"1698d6b6-be71-4474-8747-2fc8a484770a\") "
Nov 02 21:35:49.600283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.600208    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-data" pod "1698d6b6-be71-4474-8747-2fc8a484770a" (UID: "1698d6b6-be71-4474-8747-2fc8a484770a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:49.600843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.600420    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-certs" pod "1698d6b6-be71-4474-8747-2fc8a484770a" (UID: "1698d6b6-be71-4474-8747-2fc8a484770a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:49.600843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.600437    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1698d6b6-be71-4474-8747-2fc8a484770a-istiod-ca-cert" pod "1698d6b6-be71-4474-8747-2fc8a484770a" (UID: "1698d6b6-be71-4474-8747-2fc8a484770a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:35:49.600843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.600672    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-credential-socket" pod "1698d6b6-be71-4474-8747-2fc8a484770a" (UID: "1698d6b6-be71-4474-8747-2fc8a484770a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:49.601591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.601565    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:49.601718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.601593    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:49.601718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.601609    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:49.601718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.601625    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:49.601718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.601641    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1698d6b6-be71-4474-8747-2fc8a484770a-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:49.602451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.602420    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1698d6b6-be71-4474-8747-2fc8a484770a-istio-podinfo" pod "1698d6b6-be71-4474-8747-2fc8a484770a" (UID: "1698d6b6-be71-4474-8747-2fc8a484770a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:35:49.602654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.602634    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-kube-api-access-7fnmb" pod "1698d6b6-be71-4474-8747-2fc8a484770a" (UID: "1698d6b6-be71-4474-8747-2fc8a484770a"). InnerVolumeSpecName "kube-api-access-7fnmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:49.603080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.603060    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-istio-token" pod "1698d6b6-be71-4474-8747-2fc8a484770a" (UID: "1698d6b6-be71-4474-8747-2fc8a484770a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:35:49.603207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.603191    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-envoy" pod "1698d6b6-be71-4474-8747-2fc8a484770a" (UID: "1698d6b6-be71-4474-8747-2fc8a484770a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:35:49.745163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.717365    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fnmb\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-kube-api-access-7fnmb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:49.745163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.717405    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1698d6b6-be71-4474-8747-2fc8a484770a-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:49.745163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.717422    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1698d6b6-be71-4474-8747-2fc8a484770a-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:49.745163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:49.717437    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1698d6b6-be71-4474-8747-2fc8a484770a-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:35:50.244119 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.242006    4528 generic.go:356] "Generic (PLEG): container finished" podID="1698d6b6-be71-4474-8747-2fc8a484770a" containerID="6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637" exitCode=0
Nov 02 21:35:50.244119 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.242265    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" event={"ID":"1698d6b6-be71-4474-8747-2fc8a484770a","Type":"ContainerDied","Data":"6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637"}
Nov 02 21:35:50.246271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.244652    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5" event={"ID":"1698d6b6-be71-4474-8747-2fc8a484770a","Type":"ContainerDied","Data":"a61e4aeceffc858df3036b6682dca9c84788da37fe6197e09e4858e8c647fece"}
Nov 02 21:35:50.246271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.244713    4528 scope.go:122] "RemoveContainer" containerID="6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637"
Nov 02 21:35:50.246271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.244825    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"
Nov 02 21:35:50.437593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.437154    4528 scope.go:122] "RemoveContainer" containerID="6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637"
Nov 02 21:35:50.438543 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:35:50.438507    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637\": container with ID starting with 6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637 not found: ID does not exist" containerID="6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637"
Nov 02 21:35:50.438656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.438552    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637"} err="failed to get container status \"6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637\": rpc error: code = NotFound desc = could not find container \"6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637\": container with ID starting with 6fbc81e82ae300ab498a525423b0ebe847bf7e71ad04c905d6b719f8e25fc637 not found: ID does not exist"
Nov 02 21:35:50.559458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.559421    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"]
Nov 02 21:35:50.560774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.560747    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-kvhj5"]
Nov 02 21:35:50.913812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:35:50.913725    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" path="/var/lib/kubelet/pods/1698d6b6-be71-4474-8747-2fc8a484770a/volumes"
Nov 02 21:36:06.621458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.621425    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"]
Nov 02 21:36:06.622062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622044    4528 cpu_manager.go:414] "containerMap: removing container" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy"
Nov 02 21:36:06.622151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622063    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy"
Nov 02 21:36:06.622151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622075    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy"
Nov 02 21:36:06.622151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622080    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy"
Nov 02 21:36:06.622151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622088    4528 cpu_manager.go:414] "containerMap: removing container" podUID="04442c1e-13fb-4c6d-af78-72e266feed42" containerName="payload-processing"
Nov 02 21:36:06.622151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622093    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04442c1e-13fb-4c6d-af78-72e266feed42" containerName="payload-processing"
Nov 02 21:36:06.622151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622103    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy"
Nov 02 21:36:06.622151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622122    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy"
Nov 02 21:36:06.622151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622137    4528 cpu_manager.go:414] "containerMap: removing container" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy"
Nov 02 21:36:06.622151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622144    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622158    4528 cpu_manager.go:414] "containerMap: removing container" podUID="21279019-ae8a-42a2-b6c0-691809a777ea" containerName="payload-processing"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622162    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21279019-ae8a-42a2-b6c0-691809a777ea" containerName="payload-processing"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622171    4528 cpu_manager.go:414] "containerMap: removing container" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622176    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622193    4528 cpu_manager.go:414] "containerMap: removing container" podUID="970f66e4-08da-4bff-8533-1e81a9fddbb6" containerName="payload-pre-processing"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622198    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970f66e4-08da-4bff-8533-1e81a9fddbb6" containerName="payload-pre-processing"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622206    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622210    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622226    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9b0e06a0-bc8e-4cb5-b81c-b85942339178" containerName="payload-pre-processing"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622230    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0e06a0-bc8e-4cb5-b81c-b85942339178" containerName="payload-pre-processing"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622237    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e35bf9f3-4deb-417e-a3cb-7facb777aa87" containerName="revoke-keys"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622241    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35bf9f3-4deb-417e-a3cb-7facb777aa87" containerName="revoke-keys"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622341    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0e06a0-bc8e-4cb5-b81c-b85942339178" containerName="payload-pre-processing"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622351    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35bf9f3-4deb-417e-a3cb-7facb777aa87" containerName="revoke-keys"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622362    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622374    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622382    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1698d6b6-be71-4474-8747-2fc8a484770a" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622390    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="04442c1e-13fb-4c6d-af78-72e266feed42" containerName="payload-processing"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622400    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9df425-9d79-4cf9-8c80-38b4a114aba6" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622407    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="47468c66-6cd7-4d1e-a705-0689ed95559a" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622414    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="21279019-ae8a-42a2-b6c0-691809a777ea" containerName="payload-processing"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622420    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="52228fc8-539f-40bd-9f70-9c08cfed5739" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622426    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fbc8da9-f9e7-4b71-b25e-71beccf3d298" containerName="istio-proxy"
Nov 02 21:36:06.622527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.622431    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="970f66e4-08da-4bff-8533-1e81a9fddbb6" containerName="payload-pre-processing"
Nov 02 21:36:06.626188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.626159    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.678001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.677963    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"]
Nov 02 21:36:06.690146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.682343    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95300606-t8bmv\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.690146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.682433    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95300606-t8bmv\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.690146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.682470    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862k8\" (UniqueName: \"kubernetes.io/projected/887feeaa-e71a-440c-b213-f72977bbfeb5-kube-api-access-862k8\") pod \"maas-api-revoke-sub-95300606-t8bmv\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.725081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.725052    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:36:06.725081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.725065    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:36:06.725312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.725140    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:36:06.730860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.730842    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:36:06.783377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.783348    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95300606-t8bmv\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.783536 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.783409    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95300606-t8bmv\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.783536 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.783436    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862k8\" (UniqueName: \"kubernetes.io/projected/887feeaa-e71a-440c-b213-f72977bbfeb5-kube-api-access-862k8\") pod \"maas-api-revoke-sub-95300606-t8bmv\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.783955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.783930    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95300606-t8bmv\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.785863 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.785840    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95300606-t8bmv\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.840528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.840499    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862k8\" (UniqueName: \"kubernetes.io/projected/887feeaa-e71a-440c-b213-f72977bbfeb5-kube-api-access-862k8\") pod \"maas-api-revoke-sub-95300606-t8bmv\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:06.940197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:06.940098    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:07.346131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:07.333993    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"]
Nov 02 21:36:07.461150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:07.461097    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv" event={"ID":"887feeaa-e71a-440c-b213-f72977bbfeb5","Type":"ContainerStarted","Data":"efd84a106db595d13f8d5e71d01a3668ed4a5aa9a08b3a7970aa974146b28f29"}
Nov 02 21:36:07.638380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:07.638301    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:36:08.509508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:08.509422    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv" event={"ID":"887feeaa-e71a-440c-b213-f72977bbfeb5","Type":"ContainerStarted","Data":"edf0b023f763e074a26984363d71ac4dcc94d90bee55ef52ee408e16de9a9835"}
Nov 02 21:36:08.626354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:08.626301    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv" podStartSLOduration=2.626282964 podStartE2EDuration="2.626282964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:08.623743819 +0000 UTC m=+1655.887280135" watchObservedRunningTime="2026-11-02 21:36:08.626282964 +0000 UTC m=+1655.889819282"
Nov 02 21:36:09.516972 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:09.516922    4528 generic.go:356] "Generic (PLEG): container finished" podID="887feeaa-e71a-440c-b213-f72977bbfeb5" containerID="edf0b023f763e074a26984363d71ac4dcc94d90bee55ef52ee408e16de9a9835" exitCode=0
Nov 02 21:36:09.517421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:09.517007    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv" event={"ID":"887feeaa-e71a-440c-b213-f72977bbfeb5","Type":"ContainerDied","Data":"edf0b023f763e074a26984363d71ac4dcc94d90bee55ef52ee408e16de9a9835"}
Nov 02 21:36:10.753899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.753874    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:10.832588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.832522    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887feeaa-e71a-440c-b213-f72977bbfeb5-kube-api-access-862k8\" (UniqueName: \"kubernetes.io/projected/887feeaa-e71a-440c-b213-f72977bbfeb5-kube-api-access-862k8\") pod \"887feeaa-e71a-440c-b213-f72977bbfeb5\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") "
Nov 02 21:36:10.832710 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.832607    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-xks-ca\") pod \"887feeaa-e71a-440c-b213-f72977bbfeb5\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") "
Nov 02 21:36:10.832710 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.832645    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-openshift-ca\") pod \"887feeaa-e71a-440c-b213-f72977bbfeb5\" (UID: \"887feeaa-e71a-440c-b213-f72977bbfeb5\") "
Nov 02 21:36:10.833069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.833045    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-openshift-ca" pod "887feeaa-e71a-440c-b213-f72977bbfeb5" (UID: "887feeaa-e71a-440c-b213-f72977bbfeb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:36:10.833193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.833174    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:10.834571 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.834547    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-xks-ca" pod "887feeaa-e71a-440c-b213-f72977bbfeb5" (UID: "887feeaa-e71a-440c-b213-f72977bbfeb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:36:10.834677 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.834620    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887feeaa-e71a-440c-b213-f72977bbfeb5-kube-api-access-862k8" pod "887feeaa-e71a-440c-b213-f72977bbfeb5" (UID: "887feeaa-e71a-440c-b213-f72977bbfeb5"). InnerVolumeSpecName "kube-api-access-862k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:10.934534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.934508    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887feeaa-e71a-440c-b213-f72977bbfeb5-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:10.934534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:10.934534    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-862k8\" (UniqueName: \"kubernetes.io/projected/887feeaa-e71a-440c-b213-f72977bbfeb5-kube-api-access-862k8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:11.530541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:11.530509    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"
Nov 02 21:36:11.530731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:11.530512    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv" event={"ID":"887feeaa-e71a-440c-b213-f72977bbfeb5","Type":"ContainerDied","Data":"efd84a106db595d13f8d5e71d01a3668ed4a5aa9a08b3a7970aa974146b28f29"}
Nov 02 21:36:11.530731 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:11.530647    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efd84a106db595d13f8d5e71d01a3668ed4a5aa9a08b3a7970aa974146b28f29"
Nov 02 21:36:12.689339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:12.689306    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"]
Nov 02 21:36:12.690353 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:12.689538    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy" containerID="cri-o://3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c" gracePeriod=30
Nov 02 21:36:14.113308 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:14.113273    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:36:15.001030 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:15.001000    4528 patch_prober.go:29] interesting pod/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:36:15.001223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:15.001058    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:36:17.959151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:17.959090    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:36:18.018767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.018738    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-envoy\") pod \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") "
Nov 02 21:36:18.018925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.018803    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-kube-api-access-mvzj9\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-kube-api-access-mvzj9\") pod \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") "
Nov 02 21:36:18.018966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.018936    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-podinfo\") pod \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") "
Nov 02 21:36:18.018966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.018953    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istiod-ca-cert\") pod \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") "
Nov 02 21:36:18.019048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019034    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-credential-socket\") pod \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") "
Nov 02 21:36:18.019099 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019085    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-socket\") pod \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") "
Nov 02 21:36:18.019171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019128    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-data\") pod \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") "
Nov 02 21:36:18.019212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019170    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-certs\") pod \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") "
Nov 02 21:36:18.019280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019255    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-token\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-token\") pod \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\" (UID: \"dbfd76d5-b34a-41e6-badb-43f8a4a9c474\") "
Nov 02 21:36:18.019351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019320    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-credential-socket" pod "dbfd76d5-b34a-41e6-badb-43f8a4a9c474" (UID: "dbfd76d5-b34a-41e6-badb-43f8a4a9c474"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:18.019425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019402    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istiod-ca-cert" pod "dbfd76d5-b34a-41e6-badb-43f8a4a9c474" (UID: "dbfd76d5-b34a-41e6-badb-43f8a4a9c474"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:36:18.019558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019495    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-data" pod "dbfd76d5-b34a-41e6-badb-43f8a4a9c474" (UID: "dbfd76d5-b34a-41e6-badb-43f8a4a9c474"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:18.019558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019505    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-socket" pod "dbfd76d5-b34a-41e6-badb-43f8a4a9c474" (UID: "dbfd76d5-b34a-41e6-badb-43f8a4a9c474"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:18.019558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019553    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-certs" pod "dbfd76d5-b34a-41e6-badb-43f8a4a9c474" (UID: "dbfd76d5-b34a-41e6-badb-43f8a4a9c474"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:18.019999 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.019974    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:18.020164 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.020148    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:18.020284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.020262    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:18.020284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.020284    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:18.020525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.020298    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:18.021164 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.021139    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-envoy" pod "dbfd76d5-b34a-41e6-badb-43f8a4a9c474" (UID: "dbfd76d5-b34a-41e6-badb-43f8a4a9c474"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:18.021384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.021362    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-podinfo" pod "dbfd76d5-b34a-41e6-badb-43f8a4a9c474" (UID: "dbfd76d5-b34a-41e6-badb-43f8a4a9c474"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:36:18.021470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.021399    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-kube-api-access-mvzj9" pod "dbfd76d5-b34a-41e6-badb-43f8a4a9c474" (UID: "dbfd76d5-b34a-41e6-badb-43f8a4a9c474"). InnerVolumeSpecName "kube-api-access-mvzj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:18.021594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.021577    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-token" pod "dbfd76d5-b34a-41e6-badb-43f8a4a9c474" (UID: "dbfd76d5-b34a-41e6-badb-43f8a4a9c474"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:18.122079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.122042    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:18.122079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.122079    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:18.122404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.122093    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:18.122404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.122120    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvzj9\" (UniqueName: \"kubernetes.io/projected/dbfd76d5-b34a-41e6-badb-43f8a4a9c474-kube-api-access-mvzj9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:18.580155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.580119    4528 generic.go:356] "Generic (PLEG): container finished" podID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerID="3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c" exitCode=0
Nov 02 21:36:18.580328 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.580154    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" event={"ID":"dbfd76d5-b34a-41e6-badb-43f8a4a9c474","Type":"ContainerDied","Data":"3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c"}
Nov 02 21:36:18.580328 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.580177    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"
Nov 02 21:36:18.580328 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.580209    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5" event={"ID":"dbfd76d5-b34a-41e6-badb-43f8a4a9c474","Type":"ContainerDied","Data":"d0bb306842014c4fcaecc9a7dc7759c76610887426c94e6227b806b7b4058406"}
Nov 02 21:36:18.580328 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.580232    4528 scope.go:122] "RemoveContainer" containerID="3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c"
Nov 02 21:36:18.590697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.590680    4528 scope.go:122] "RemoveContainer" containerID="3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7"
Nov 02 21:36:18.601047 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.601027    4528 scope.go:122] "RemoveContainer" containerID="3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c"
Nov 02 21:36:18.601309 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:18.601289    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c\": container with ID starting with 3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c not found: ID does not exist" containerID="3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c"
Nov 02 21:36:18.601371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.601317    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c"} err="failed to get container status \"3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c\": rpc error: code = NotFound desc = could not find container \"3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c\": container with ID starting with 3208a95cf9a9239672779ccfd80770b74af0fd9ae1fb9a44ff914ca9d5b6b47c not found: ID does not exist"
Nov 02 21:36:18.601371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.601334    4528 scope.go:122] "RemoveContainer" containerID="3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7"
Nov 02 21:36:18.601548 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:18.601532    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7\": container with ID starting with 3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7 not found: ID does not exist" containerID="3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7"
Nov 02 21:36:18.601600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.601552    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7"} err="failed to get container status \"3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7\": rpc error: code = NotFound desc = could not find container \"3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7\": container with ID starting with 3d63fffd004c783a4a368e812d2af2f87bc9eeeb2fbc657880df8c2735c66be7 not found: ID does not exist"
Nov 02 21:36:18.746177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.746130    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"]
Nov 02 21:36:18.751796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.751769    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-54a4b2-openshift-default-69988bcdcf-qdfn5"]
Nov 02 21:36:18.913326 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:18.913248    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" path="/var/lib/kubelet/pods/dbfd76d5-b34a-41e6-badb-43f8a4a9c474/volumes"
Nov 02 21:36:26.939801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.939760    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"]
Nov 02 21:36:26.940321 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.940308    4528 cpu_manager.go:414] "containerMap: removing container" podUID="887feeaa-e71a-440c-b213-f72977bbfeb5" containerName="revoke-keys"
Nov 02 21:36:26.940321 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.940322    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887feeaa-e71a-440c-b213-f72977bbfeb5" containerName="revoke-keys"
Nov 02 21:36:26.940428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.940335    4528 cpu_manager.go:414] "containerMap: removing container" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy"
Nov 02 21:36:26.940428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.940340    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy"
Nov 02 21:36:26.940428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.940347    4528 cpu_manager.go:414] "containerMap: removing container" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy"
Nov 02 21:36:26.940428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.940352    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ed76ea-1ce8-4b37-b55c-5c98e5eef187" containerName="istio-proxy"
Nov 02 21:36:26.940552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.940458    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="887feeaa-e71a-440c-b213-f72977bbfeb5" containerName="revoke-keys"
Nov 02 21:36:26.940552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.940467    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy"
Nov 02 21:36:26.940552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.940478    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy"
Nov 02 21:36:26.943762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:26.943746    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:27.000686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.000632    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"]
Nov 02 21:36:27.043545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.043509    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-54e844\""
Nov 02 21:36:27.043972 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.043895    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-54e844-dockercfg-n48lp\""
Nov 02 21:36:27.258265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.187234    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ffd75a-0fa0-4960-8f10-4815301e82eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2\" (UID: \"21ffd75a-0fa0-4960-8f10-4815301e82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:27.258265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.187314    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8rfd\" (UniqueName: \"kubernetes.io/projected/21ffd75a-0fa0-4960-8f10-4815301e82eb-kube-api-access-t8rfd\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2\" (UID: \"21ffd75a-0fa0-4960-8f10-4815301e82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:27.268239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.268207    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"]
Nov 02 21:36:27.269432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.269407    4528 cpu_manager.go:414] "containerMap: removing container" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy"
Nov 02 21:36:27.269822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.269802    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfd76d5-b34a-41e6-badb-43f8a4a9c474" containerName="istio-proxy"
Nov 02 21:36:27.277210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.276155    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"]
Nov 02 21:36:27.277210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.276271    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:27.288725 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.288692    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ffd75a-0fa0-4960-8f10-4815301e82eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2\" (UID: \"21ffd75a-0fa0-4960-8f10-4815301e82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:27.288849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.288761    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8rfd\" (UniqueName: \"kubernetes.io/projected/21ffd75a-0fa0-4960-8f10-4815301e82eb-kube-api-access-t8rfd\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2\" (UID: \"21ffd75a-0fa0-4960-8f10-4815301e82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:27.289994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.289941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ffd75a-0fa0-4960-8f10-4815301e82eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2\" (UID: \"21ffd75a-0fa0-4960-8f10-4815301e82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:27.345984 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.345952    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8rfd\" (UniqueName: \"kubernetes.io/projected/21ffd75a-0fa0-4960-8f10-4815301e82eb-kube-api-access-t8rfd\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2\" (UID: \"21ffd75a-0fa0-4960-8f10-4815301e82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:27.389882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.389839    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qct74\" (UniqueName: \"kubernetes.io/projected/97cad693-11bb-4797-b96f-5a966171a90c-kube-api-access-qct74\") pod \"payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d\" (UID: \"97cad693-11bb-4797-b96f-5a966171a90c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:27.390212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.390179    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cad693-11bb-4797-b96f-5a966171a90c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d\" (UID: \"97cad693-11bb-4797-b96f-5a966171a90c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:27.493341 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.493283    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cad693-11bb-4797-b96f-5a966171a90c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d\" (UID: \"97cad693-11bb-4797-b96f-5a966171a90c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:27.493766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.493745    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qct74\" (UniqueName: \"kubernetes.io/projected/97cad693-11bb-4797-b96f-5a966171a90c-kube-api-access-qct74\") pod \"payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d\" (UID: \"97cad693-11bb-4797-b96f-5a966171a90c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:27.495307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.495269    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cad693-11bb-4797-b96f-5a966171a90c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d\" (UID: \"97cad693-11bb-4797-b96f-5a966171a90c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:27.551523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.551487    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qct74\" (UniqueName: \"kubernetes.io/projected/97cad693-11bb-4797-b96f-5a966171a90c-kube-api-access-qct74\") pod \"payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d\" (UID: \"97cad693-11bb-4797-b96f-5a966171a90c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:27.557355 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.557324    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:27.580127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.580084    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"]
Nov 02 21:36:27.580427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.580394    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy" containerID="cri-o://cb487527b8df7540a004c089458bb2195c5bca3a62f9d8ddddc06a19f74b9961" gracePeriod=30
Nov 02 21:36:27.586383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.586356    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"]
Nov 02 21:36:27.586655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.586623    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy" containerID="cri-o://b2c20a520ca5137b499077c762d67f69390391ca1bc5c49875d3307fdf96d591" gracePeriod=30
Nov 02 21:36:27.595127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.594455    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:27.950432 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:36:27.950401    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ffd75a_0fa0_4960_8f10_4815301e82eb.slice/crio-da6bb86feaeb54e43b5c5e758b158adf4609abb0802e487905f02c1e968db256 WatchSource:0}: Error finding container da6bb86feaeb54e43b5c5e758b158adf4609abb0802e487905f02c1e968db256: Status 404 returned error can't find the container with id da6bb86feaeb54e43b5c5e758b158adf4609abb0802e487905f02c1e968db256
Nov 02 21:36:27.952783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:27.952757    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"]
Nov 02 21:36:28.010021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:28.009997    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"]
Nov 02 21:36:28.010381 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:36:28.010351    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cad693_11bb_4797_b96f_5a966171a90c.slice/crio-fde519b18c79ed33f232e244e5bdb717bed3334b4cce704a87f61f8705c3000e WatchSource:0}: Error finding container fde519b18c79ed33f232e244e5bdb717bed3334b4cce704a87f61f8705c3000e: Status 404 returned error can't find the container with id fde519b18c79ed33f232e244e5bdb717bed3334b4cce704a87f61f8705c3000e
Nov 02 21:36:28.877785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:28.877742    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2" event={"ID":"21ffd75a-0fa0-4960-8f10-4815301e82eb","Type":"ContainerStarted","Data":"5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9"}
Nov 02 21:36:28.877978 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:28.877817    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2" event={"ID":"21ffd75a-0fa0-4960-8f10-4815301e82eb","Type":"ContainerStarted","Data":"da6bb86feaeb54e43b5c5e758b158adf4609abb0802e487905f02c1e968db256"}
Nov 02 21:36:28.877978 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:28.877852    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:28.879618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:28.879590    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d" event={"ID":"97cad693-11bb-4797-b96f-5a966171a90c","Type":"ContainerStarted","Data":"b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a"}
Nov 02 21:36:28.879762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:28.879644    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d" event={"ID":"97cad693-11bb-4797-b96f-5a966171a90c","Type":"ContainerStarted","Data":"fde519b18c79ed33f232e244e5bdb717bed3334b4cce704a87f61f8705c3000e"}
Nov 02 21:36:28.879762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:28.879685    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:28.994333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:28.994275    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2" podStartSLOduration=2.99426109 podStartE2EDuration="2.99426109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:28.99050849 +0000 UTC m=+1676.254044817" watchObservedRunningTime="2026-11-02 21:36:28.99426109 +0000 UTC m=+1676.257797407"
Nov 02 21:36:32.930971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:32.930938    4528 generic.go:356] "Generic (PLEG): container finished" podID="0355ee02-9c37-473d-9555-e60358738c22" containerID="b2c20a520ca5137b499077c762d67f69390391ca1bc5c49875d3307fdf96d591" exitCode=0
Nov 02 21:36:32.931326 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:32.931008    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" event={"ID":"0355ee02-9c37-473d-9555-e60358738c22","Type":"ContainerDied","Data":"b2c20a520ca5137b499077c762d67f69390391ca1bc5c49875d3307fdf96d591"}
Nov 02 21:36:32.932442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:32.932422    4528 generic.go:356] "Generic (PLEG): container finished" podID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerID="cb487527b8df7540a004c089458bb2195c5bca3a62f9d8ddddc06a19f74b9961" exitCode=0
Nov 02 21:36:32.932526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:32.932490    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" event={"ID":"9d258233-0e4a-48ae-a387-94c13aa4c401","Type":"ContainerDied","Data":"cb487527b8df7540a004c089458bb2195c5bca3a62f9d8ddddc06a19f74b9961"}
Nov 02 21:36:32.932585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:32.932526    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp" event={"ID":"9d258233-0e4a-48ae-a387-94c13aa4c401","Type":"ContainerDied","Data":"0461f96b026082c7e213f2096df87d50376a8a69d3d96637901a145815cf65ea"}
Nov 02 21:36:32.932585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:32.932542    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0461f96b026082c7e213f2096df87d50376a8a69d3d96637901a145815cf65ea"
Nov 02 21:36:32.959331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:32.959308    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:36:32.962663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:32.962645    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:36:33.107875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.107841    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-kube-api-access-6chql\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-kube-api-access-6chql\") pod \"9d258233-0e4a-48ae-a387-94c13aa4c401\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") "
Nov 02 21:36:33.108166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.108143    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d258233-0e4a-48ae-a387-94c13aa4c401-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d258233-0e4a-48ae-a387-94c13aa4c401-istiod-ca-cert\") pod \"9d258233-0e4a-48ae-a387-94c13aa4c401\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") "
Nov 02 21:36:33.108323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.108307    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-socket\") pod \"0355ee02-9c37-473d-9555-e60358738c22\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") "
Nov 02 21:36:33.108549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.108527    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-data\") pod \"9d258233-0e4a-48ae-a387-94c13aa4c401\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") "
Nov 02 21:36:33.108984 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.108881    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-kube-api-access-6s9rk\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-kube-api-access-6s9rk\") pod \"0355ee02-9c37-473d-9555-e60358738c22\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") "
Nov 02 21:36:33.109244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.109216    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-socket\") pod \"9d258233-0e4a-48ae-a387-94c13aa4c401\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") "
Nov 02 21:36:33.109416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.109391    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-istio-token\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-istio-token\") pod \"0355ee02-9c37-473d-9555-e60358738c22\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") "
Nov 02 21:36:33.109580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.109551    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-credential-socket\") pod \"9d258233-0e4a-48ae-a387-94c13aa4c401\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") "
Nov 02 21:36:33.109784 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.109767    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-token\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-token\") pod \"9d258233-0e4a-48ae-a387-94c13aa4c401\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") "
Nov 02 21:36:33.109986 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.109950    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-certs\") pod \"0355ee02-9c37-473d-9555-e60358738c22\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") "
Nov 02 21:36:33.110146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.110130    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-podinfo\") pod \"9d258233-0e4a-48ae-a387-94c13aa4c401\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") "
Nov 02 21:36:33.110283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.110268    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-certs\") pod \"9d258233-0e4a-48ae-a387-94c13aa4c401\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") "
Nov 02 21:36:33.110467 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.110453    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-envoy\") pod \"9d258233-0e4a-48ae-a387-94c13aa4c401\" (UID: \"9d258233-0e4a-48ae-a387-94c13aa4c401\") "
Nov 02 21:36:33.114232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.114193    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0355ee02-9c37-473d-9555-e60358738c22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0355ee02-9c37-473d-9555-e60358738c22-istio-podinfo\") pod \"0355ee02-9c37-473d-9555-e60358738c22\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") "
Nov 02 21:36:33.114508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.114480    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-data\") pod \"0355ee02-9c37-473d-9555-e60358738c22\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") "
Nov 02 21:36:33.118199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.118163    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0355ee02-9c37-473d-9555-e60358738c22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0355ee02-9c37-473d-9555-e60358738c22-istiod-ca-cert\") pod \"0355ee02-9c37-473d-9555-e60358738c22\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") "
Nov 02 21:36:33.118367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.118351    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-envoy\") pod \"0355ee02-9c37-473d-9555-e60358738c22\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") "
Nov 02 21:36:33.118493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.118478    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-credential-socket\") pod \"0355ee02-9c37-473d-9555-e60358738c22\" (UID: \"0355ee02-9c37-473d-9555-e60358738c22\") "
Nov 02 21:36:33.162149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.161788    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-data" pod "9d258233-0e4a-48ae-a387-94c13aa4c401" (UID: "9d258233-0e4a-48ae-a387-94c13aa4c401"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.175259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.162415    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-certs" pod "0355ee02-9c37-473d-9555-e60358738c22" (UID: "0355ee02-9c37-473d-9555-e60358738c22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.175259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.165295    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-socket" pod "9d258233-0e4a-48ae-a387-94c13aa4c401" (UID: "9d258233-0e4a-48ae-a387-94c13aa4c401"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.176782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.175730    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-certs" pod "9d258233-0e4a-48ae-a387-94c13aa4c401" (UID: "9d258233-0e4a-48ae-a387-94c13aa4c401"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.176782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.176015    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-credential-socket" pod "9d258233-0e4a-48ae-a387-94c13aa4c401" (UID: "9d258233-0e4a-48ae-a387-94c13aa4c401"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.176782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.176329    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d258233-0e4a-48ae-a387-94c13aa4c401-istiod-ca-cert" pod "9d258233-0e4a-48ae-a387-94c13aa4c401" (UID: "9d258233-0e4a-48ae-a387-94c13aa4c401"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:36:33.176782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.176534    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-socket" pod "0355ee02-9c37-473d-9555-e60358738c22" (UID: "0355ee02-9c37-473d-9555-e60358738c22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.185398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.185339    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d" podStartSLOduration=6.185317937 podStartE2EDuration="6.185317937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:29.117622481 +0000 UTC m=+1676.381158824" watchObservedRunningTime="2026-11-02 21:36:33.185317937 +0000 UTC m=+1680.448854257"
Nov 02 21:36:33.189577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.189539    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-credential-socket" pod "0355ee02-9c37-473d-9555-e60358738c22" (UID: "0355ee02-9c37-473d-9555-e60358738c22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.189902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.189877    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0355ee02-9c37-473d-9555-e60358738c22-istiod-ca-cert" pod "0355ee02-9c37-473d-9555-e60358738c22" (UID: "0355ee02-9c37-473d-9555-e60358738c22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:36:33.190313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.190283    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-data" pod "0355ee02-9c37-473d-9555-e60358738c22" (UID: "0355ee02-9c37-473d-9555-e60358738c22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.245518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.245481    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-envoy" pod "0355ee02-9c37-473d-9555-e60358738c22" (UID: "0355ee02-9c37-473d-9555-e60358738c22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.275169 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.245777    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0355ee02-9c37-473d-9555-e60358738c22-istio-podinfo" pod "0355ee02-9c37-473d-9555-e60358738c22" (UID: "0355ee02-9c37-473d-9555-e60358738c22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:36:33.275386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.245828    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-envoy" pod "9d258233-0e4a-48ae-a387-94c13aa4c401" (UID: "9d258233-0e4a-48ae-a387-94c13aa4c401"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:33.275481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.258725    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-kube-api-access-6s9rk" pod "0355ee02-9c37-473d-9555-e60358738c22" (UID: "0355ee02-9c37-473d-9555-e60358738c22"). InnerVolumeSpecName "kube-api-access-6s9rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:33.275558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.259004    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-podinfo" pod "9d258233-0e4a-48ae-a387-94c13aa4c401" (UID: "9d258233-0e4a-48ae-a387-94c13aa4c401"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:36:33.280250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.279710    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-kube-api-access-6chql" pod "9d258233-0e4a-48ae-a387-94c13aa4c401" (UID: "9d258233-0e4a-48ae-a387-94c13aa4c401"). InnerVolumeSpecName "kube-api-access-6chql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:33.280374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.265892    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.280476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.280464    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.280532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.280524    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.280580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.280574    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.280629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.280622    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0355ee02-9c37-473d-9555-e60358738c22-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.280682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.280675    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.280735 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.280728    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d258233-0e4a-48ae-a387-94c13aa4c401-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.280784 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.280777    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.280833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.280826    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.280881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.280873    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.281317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.281133    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-token" pod "9d258233-0e4a-48ae-a387-94c13aa4c401" (UID: "9d258233-0e4a-48ae-a387-94c13aa4c401"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:33.281533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.281501    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-istio-token" pod "0355ee02-9c37-473d-9555-e60358738c22" (UID: "0355ee02-9c37-473d-9555-e60358738c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:33.384000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.382887    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.384000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.382932    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0355ee02-9c37-473d-9555-e60358738c22-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.384000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.382948    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0355ee02-9c37-473d-9555-e60358738c22-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.384000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.382962    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6chql\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-kube-api-access-6chql\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.384000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.382977    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s9rk\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-kube-api-access-6s9rk\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.384000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.382989    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0355ee02-9c37-473d-9555-e60358738c22-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.384000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.383001    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.384000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.383014    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d258233-0e4a-48ae-a387-94c13aa4c401-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:33.765130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.764622    4528 scope.go:122] "RemoveContainer" containerID="b4d46866061ac7a52784a7b4d978481b74345714e449f98f8a16ba12b7a61c04"
Nov 02 21:36:33.824855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.824830    4528 scope.go:122] "RemoveContainer" containerID="a66ca22ee8656b6758fdf10a1b7cff46a3d69edd9077701366ac0e77a30299a6"
Nov 02 21:36:33.836003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.835976    4528 scope.go:122] "RemoveContainer" containerID="a45b29ac0aca8f5a1057dba8e8dfdef6722adee6fee4dee75e8add3187a39530"
Nov 02 21:36:33.845012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.844986    4528 scope.go:122] "RemoveContainer" containerID="af83187215c733ad711a9320f640dbd8b92ae149cff8fb41bd29fc174739582e"
Nov 02 21:36:33.854090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.854066    4528 scope.go:122] "RemoveContainer" containerID="9ce150e9d4f9d2219537f17e889bff272f11fdbf7a79749bd7322fd8ff5d5714"
Nov 02 21:36:33.863811 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.863783    4528 scope.go:122] "RemoveContainer" containerID="97ea87470bd7a38b6fdec1465ab345689a2d09cf7bfc2624dc6460bddc3a3b22"
Nov 02 21:36:33.874018 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.873991    4528 scope.go:122] "RemoveContainer" containerID="009b5fd45047372c2fb0b3c91774e0d853a6f3c6f9a57e69f112f40d3a37c0b6"
Nov 02 21:36:33.883546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.883523    4528 scope.go:122] "RemoveContainer" containerID="9c0669bf0835586c12ca73ffba353c35d13da0a0ef2eca8e77ad237254b89d67"
Nov 02 21:36:33.894721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.894696    4528 scope.go:122] "RemoveContainer" containerID="93104cb5c471788a2328e1c85dac92fd665940709c676524a346a0a1b7af9a94"
Nov 02 21:36:33.904286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.904265    4528 scope.go:122] "RemoveContainer" containerID="ac4d62a800edbb22fc4baee46ea19d1b3c59f8d2d8d433c1a5dd9a20a3747841"
Nov 02 21:36:33.912468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.912443    4528 scope.go:122] "RemoveContainer" containerID="cb487527b8df7540a004c089458bb2195c5bca3a62f9d8ddddc06a19f74b9961"
Nov 02 21:36:33.973692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.973634    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"
Nov 02 21:36:33.975518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.975483    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c" event={"ID":"0355ee02-9c37-473d-9555-e60358738c22","Type":"ContainerDied","Data":"38cba048a90bcd60f02bff82106d5a1ad2356dff199bca37f42262337b38322c"}
Nov 02 21:36:33.975627 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.975553    4528 scope.go:122] "RemoveContainer" containerID="b2c20a520ca5137b499077c762d67f69390391ca1bc5c49875d3307fdf96d591"
Nov 02 21:36:33.977173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:33.977152    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"
Nov 02 21:36:34.162267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:34.160691    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"]
Nov 02 21:36:34.203552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:34.203471    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-rzjkp"]
Nov 02 21:36:34.273433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:34.273400    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"]
Nov 02 21:36:34.277954 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:34.277929    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-lfd6c"]
Nov 02 21:36:34.913357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:34.913327    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0355ee02-9c37-473d-9555-e60358738c22" path="/var/lib/kubelet/pods/0355ee02-9c37-473d-9555-e60358738c22/volumes"
Nov 02 21:36:34.913787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:34.913773    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" path="/var/lib/kubelet/pods/9d258233-0e4a-48ae-a387-94c13aa4c401/volumes"
Nov 02 21:36:38.685721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:38.685685    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"]
Nov 02 21:36:38.686292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:38.686015    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy" containerID="cri-o://7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e" gracePeriod=30
Nov 02 21:36:39.904532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:39.904503    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:36:39.905019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:39.904572    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:36:40.152975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.152928    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"]
Nov 02 21:36:40.153531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.153516    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy"
Nov 02 21:36:40.153585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.153533    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy"
Nov 02 21:36:40.153585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.153576    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy"
Nov 02 21:36:40.153585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.153581    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy"
Nov 02 21:36:40.153690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.153659    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0355ee02-9c37-473d-9555-e60358738c22" containerName="istio-proxy"
Nov 02 21:36:40.153690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.153668    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d258233-0e4a-48ae-a387-94c13aa4c401" containerName="istio-proxy"
Nov 02 21:36:40.159077 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.159024    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.235656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.235625    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-15253e-openshift-default-dockercfg-swdsp\""
Nov 02 21:36:40.278539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.278502    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8522m\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-kube-api-access-8522m\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.278900 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.278874    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-certs\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.279058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.279042    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-data\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.279207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.279190    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971579a-0d19-4119-9924-df6165cfb992-istio-podinfo\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.279372 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.279350    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-envoy\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.279534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.279516    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-credential-socket\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.291173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.283176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971579a-0d19-4119-9924-df6165cfb992-istiod-ca-cert\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.314135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.291453    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-istio-token\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.331685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.331648    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-socket\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.385883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.385796    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"]
Nov 02 21:36:40.453198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.439904    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-certs\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.453198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.439980    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-data\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.453198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.440020    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971579a-0d19-4119-9924-df6165cfb992-istio-podinfo\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.453198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.440070    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-envoy\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.453198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.440138    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-credential-socket\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.453198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.440172    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971579a-0d19-4119-9924-df6165cfb992-istiod-ca-cert\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.453198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.440210    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-istio-token\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.453198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.440250    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-socket\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.453198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.440343    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8522m\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-kube-api-access-8522m\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.461164 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.460270    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-credential-socket\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.461164 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.460727    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-data\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.462163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.462137    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"]
Nov 02 21:36:40.462625 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.462577    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy" containerID="cri-o://fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6" gracePeriod=30
Nov 02 21:36:40.463657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.462881    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-socket\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.463657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.463011    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971579a-0d19-4119-9924-df6165cfb992-istiod-ca-cert\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.480266 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.480228    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-certs\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.519780 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.519735    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971579a-0d19-4119-9924-df6165cfb992-istio-podinfo\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.519961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.519804    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"]
Nov 02 21:36:40.519961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.519859    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"]
Nov 02 21:36:40.520091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.520019    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy" containerID="cri-o://942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224" gracePeriod=30
Nov 02 21:36:40.520436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.520353    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy" containerID="cri-o://1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2" gracePeriod=30
Nov 02 21:36:40.522058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.521976    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-envoy\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.532137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.532058    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"]
Nov 02 21:36:40.532306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.532275    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy" containerID="cri-o://6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a" gracePeriod=30
Nov 02 21:36:40.535573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.535501    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8522m\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-kube-api-access-8522m\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.540662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.540620    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-istio-token\") pod \"e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") " pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:40.774291 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:40.774257    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:41.197388 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:36:41.197357    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf971579a_0d19_4119_9924_df6165cfb992.slice/crio-86d381f62fdc2305eed3fb0174bc89dca604f3d814f8e51c85ae62c5862ef67b WatchSource:0}: Error finding container 86d381f62fdc2305eed3fb0174bc89dca604f3d814f8e51c85ae62c5862ef67b: Status 404 returned error can't find the container with id 86d381f62fdc2305eed3fb0174bc89dca604f3d814f8e51c85ae62c5862ef67b
Nov 02 21:36:41.199689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:41.199665    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"]
Nov 02 21:36:41.199971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:41.199940    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:36:41.200048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:41.200016    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:36:41.200139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:41.200059    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:36:42.070061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:42.070022    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" event={"ID":"f971579a-0d19-4119-9924-df6165cfb992","Type":"ContainerStarted","Data":"e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898"}
Nov 02 21:36:42.070266 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:42.070087    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" event={"ID":"f971579a-0d19-4119-9924-df6165cfb992","Type":"ContainerStarted","Data":"86d381f62fdc2305eed3fb0174bc89dca604f3d814f8e51c85ae62c5862ef67b"}
Nov 02 21:36:42.128618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:42.128588    4528 patch_prober.go:29] interesting pod/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:36:42.128766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:42.128638    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:36:42.289008 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:42.288935    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" podStartSLOduration=2.288920696 podStartE2EDuration="2.288920696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:42.235938416 +0000 UTC m=+1689.499474732" watchObservedRunningTime="2026-11-02 21:36:42.288920696 +0000 UTC m=+1689.552457014"
Nov 02 21:36:42.775381 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:42.775348    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:42.780546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:42.780522    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:43.076606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:43.076518    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:43.077499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:43.077481    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:36:43.956788 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:43.956733    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:36:44.083633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.083600    4528 generic.go:356] "Generic (PLEG): container finished" podID="9d8bd196-f730-46cd-aa00-23352f984e61" containerID="7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e" exitCode=0
Nov 02 21:36:44.083830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.083653    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"
Nov 02 21:36:44.083830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.083689    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" event={"ID":"9d8bd196-f730-46cd-aa00-23352f984e61","Type":"ContainerDied","Data":"7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e"}
Nov 02 21:36:44.083830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.083780    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g" event={"ID":"9d8bd196-f730-46cd-aa00-23352f984e61","Type":"ContainerDied","Data":"feaf1c6e442aedf71971a6b3854094d68dd1f352042ac8212ebe5d5e527b82d0"}
Nov 02 21:36:44.083830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.083809    4528 scope.go:122] "RemoveContainer" containerID="7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e"
Nov 02 21:36:44.089526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089505    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-certs\") pod \"9d8bd196-f730-46cd-aa00-23352f984e61\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") "
Nov 02 21:36:44.089634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089583    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-envoy\") pod \"9d8bd196-f730-46cd-aa00-23352f984e61\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") "
Nov 02 21:36:44.089698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089631    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-socket\") pod \"9d8bd196-f730-46cd-aa00-23352f984e61\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") "
Nov 02 21:36:44.089698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089660    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8bd196-f730-46cd-aa00-23352f984e61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8bd196-f730-46cd-aa00-23352f984e61-istiod-ca-cert\") pod \"9d8bd196-f730-46cd-aa00-23352f984e61\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") "
Nov 02 21:36:44.089780 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089736    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-kube-api-access-8tpvq\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-kube-api-access-8tpvq\") pod \"9d8bd196-f730-46cd-aa00-23352f984e61\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") "
Nov 02 21:36:44.089833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089781    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-data\") pod \"9d8bd196-f730-46cd-aa00-23352f984e61\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") "
Nov 02 21:36:44.089833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089814    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-certs" pod "9d8bd196-f730-46cd-aa00-23352f984e61" (UID: "9d8bd196-f730-46cd-aa00-23352f984e61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:44.089928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089843    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-credential-socket\") pod \"9d8bd196-f730-46cd-aa00-23352f984e61\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") "
Nov 02 21:36:44.089928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089868    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-istio-token\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-istio-token\") pod \"9d8bd196-f730-46cd-aa00-23352f984e61\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") "
Nov 02 21:36:44.089928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.089904    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d8bd196-f730-46cd-aa00-23352f984e61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8bd196-f730-46cd-aa00-23352f984e61-istio-podinfo\") pod \"9d8bd196-f730-46cd-aa00-23352f984e61\" (UID: \"9d8bd196-f730-46cd-aa00-23352f984e61\") "
Nov 02 21:36:44.090155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.090132    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-credential-socket" pod "9d8bd196-f730-46cd-aa00-23352f984e61" (UID: "9d8bd196-f730-46cd-aa00-23352f984e61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:44.090248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.090223    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8bd196-f730-46cd-aa00-23352f984e61-istiod-ca-cert" pod "9d8bd196-f730-46cd-aa00-23352f984e61" (UID: "9d8bd196-f730-46cd-aa00-23352f984e61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:36:44.090490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.090469    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-socket" pod "9d8bd196-f730-46cd-aa00-23352f984e61" (UID: "9d8bd196-f730-46cd-aa00-23352f984e61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:44.090690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.090667    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d8bd196-f730-46cd-aa00-23352f984e61-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:44.090690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.090682    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-data" pod "9d8bd196-f730-46cd-aa00-23352f984e61" (UID: "9d8bd196-f730-46cd-aa00-23352f984e61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:44.090690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.090698    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:44.090948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.090714    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:44.092092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.092050    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-envoy" pod "9d8bd196-f730-46cd-aa00-23352f984e61" (UID: "9d8bd196-f730-46cd-aa00-23352f984e61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:44.092092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.092068    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d8bd196-f730-46cd-aa00-23352f984e61-istio-podinfo" pod "9d8bd196-f730-46cd-aa00-23352f984e61" (UID: "9d8bd196-f730-46cd-aa00-23352f984e61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:36:44.092623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.092600    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-istio-token" pod "9d8bd196-f730-46cd-aa00-23352f984e61" (UID: "9d8bd196-f730-46cd-aa00-23352f984e61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:44.092712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.092696    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-kube-api-access-8tpvq" pod "9d8bd196-f730-46cd-aa00-23352f984e61" (UID: "9d8bd196-f730-46cd-aa00-23352f984e61"). InnerVolumeSpecName "kube-api-access-8tpvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:44.095723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.095699    4528 scope.go:122] "RemoveContainer" containerID="1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545"
Nov 02 21:36:44.114611 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:44.114582    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:36:44.117954 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.117934    4528 scope.go:122] "RemoveContainer" containerID="7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e"
Nov 02 21:36:44.118249 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:44.118229    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e\": container with ID starting with 7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e not found: ID does not exist" containerID="7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e"
Nov 02 21:36:44.118303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.118259    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e"} err="failed to get container status \"7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e\": rpc error: code = NotFound desc = could not find container \"7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e\": container with ID starting with 7ddc6f17075df5aa5fc733ada354d268ff90700d55ddcfd552c254002891c59e not found: ID does not exist"
Nov 02 21:36:44.118303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.118276    4528 scope.go:122] "RemoveContainer" containerID="1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545"
Nov 02 21:36:44.118516 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:44.118496    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545\": container with ID starting with 1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545 not found: ID does not exist" containerID="1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545"
Nov 02 21:36:44.118577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.118526    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545"} err="failed to get container status \"1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545\": rpc error: code = NotFound desc = could not find container \"1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545\": container with ID starting with 1698dee8dc1beefc1b402808eb7cc8634b8d9a6294e5217f2741560cb9312545 not found: ID does not exist"
Nov 02 21:36:44.192562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.192527    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:44.192562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.192556    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:44.192562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.192570    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tpvq\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-kube-api-access-8tpvq\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:44.192800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.192583    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d8bd196-f730-46cd-aa00-23352f984e61-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:44.192800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.192594    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d8bd196-f730-46cd-aa00-23352f984e61-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:44.192800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.192605    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d8bd196-f730-46cd-aa00-23352f984e61-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:44.506628 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.506572    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"]
Nov 02 21:36:44.509315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.509295    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bd96ef-openshift-default-7447fc897f-8885g"]
Nov 02 21:36:44.914497 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:44.914464    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" path="/var/lib/kubelet/pods/9d8bd196-f730-46cd-aa00-23352f984e61/volumes"
Nov 02 21:36:45.076726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.076693    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:36:45.077126 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.076752    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:36:45.760887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.760860    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:36:45.800580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.800557    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:36:45.815363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.815340    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:36:45.817598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817573    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/409ba5bd-00c3-401d-bbda-e6191877887f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409ba5bd-00c3-401d-bbda-e6191877887f-istio-podinfo\") pod \"409ba5bd-00c3-401d-bbda-e6191877887f\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") "
Nov 02 21:36:45.817746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817613    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-envoy\") pod \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") "
Nov 02 21:36:45.817746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817651    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-kube-api-access-2h5pw\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-kube-api-access-2h5pw\") pod \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") "
Nov 02 21:36:45.817746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817693    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-data\") pod \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") "
Nov 02 21:36:45.817746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817734    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409ba5bd-00c3-401d-bbda-e6191877887f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409ba5bd-00c3-401d-bbda-e6191877887f-istiod-ca-cert\") pod \"409ba5bd-00c3-401d-bbda-e6191877887f\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817825    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-socket\") pod \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817855    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-token\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-token\") pod \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817887    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-credential-socket\") pod \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817915    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-podinfo\") pod \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817940    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istiod-ca-cert\") pod \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.817977    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istiod-ca-cert\") pod \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818026    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-envoy\") pod \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818076    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-credential-socket\") pod \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818133    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-certs\") pod \"409ba5bd-00c3-401d-bbda-e6191877887f\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818223    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-certs\") pod \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818260    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-certs\") pod \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818290    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-socket\") pod \"409ba5bd-00c3-401d-bbda-e6191877887f\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818317    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-istio-token\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-istio-token\") pod \"409ba5bd-00c3-401d-bbda-e6191877887f\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818339    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409ba5bd-00c3-401d-bbda-e6191877887f-istiod-ca-cert" pod "409ba5bd-00c3-401d-bbda-e6191877887f" (UID: "409ba5bd-00c3-401d-bbda-e6191877887f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818345    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-credential-socket\") pod \"409ba5bd-00c3-401d-bbda-e6191877887f\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818391    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-data\") pod \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") "
Nov 02 21:36:45.818452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818421    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-socket\") pod \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") "
Nov 02 21:36:45.819221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818467    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-podinfo\") pod \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\" (UID: \"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8\") "
Nov 02 21:36:45.819221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818511    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-kube-api-access-tr9n6\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-kube-api-access-tr9n6\") pod \"409ba5bd-00c3-401d-bbda-e6191877887f\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") "
Nov 02 21:36:45.819221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818551    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-envoy\") pod \"409ba5bd-00c3-401d-bbda-e6191877887f\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") "
Nov 02 21:36:45.819221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818622    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-token\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-token\") pod \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") "
Nov 02 21:36:45.819221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818654    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-data\") pod \"409ba5bd-00c3-401d-bbda-e6191877887f\" (UID: \"409ba5bd-00c3-401d-bbda-e6191877887f\") "
Nov 02 21:36:45.819221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.818690    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-kube-api-access-q4dw8\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-kube-api-access-q4dw8\") pod \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\" (UID: \"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c\") "
Nov 02 21:36:45.819482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.819386    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409ba5bd-00c3-401d-bbda-e6191877887f-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.820771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.820535    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istiod-ca-cert" pod "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" (UID: "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:36:45.821092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.820341    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-socket" pod "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" (UID: "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.821092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.821059    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-data" pod "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" (UID: "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.821284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.821190    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-envoy" pod "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" (UID: "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.822322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.822072    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istiod-ca-cert" pod "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" (UID: "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:36:45.822422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.822380    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/409ba5bd-00c3-401d-bbda-e6191877887f-istio-podinfo" pod "409ba5bd-00c3-401d-bbda-e6191877887f" (UID: "409ba5bd-00c3-401d-bbda-e6191877887f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:36:45.822667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.822594    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-credential-socket" pod "409ba5bd-00c3-401d-bbda-e6191877887f" (UID: "409ba5bd-00c3-401d-bbda-e6191877887f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.822757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.822688    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-data" pod "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" (UID: "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.822888 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.822862    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-credential-socket" pod "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" (UID: "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.823837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.823812    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-data" pod "409ba5bd-00c3-401d-bbda-e6191877887f" (UID: "409ba5bd-00c3-401d-bbda-e6191877887f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.824191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.824135    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-certs" pod "409ba5bd-00c3-401d-bbda-e6191877887f" (UID: "409ba5bd-00c3-401d-bbda-e6191877887f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.824309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.824267    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-envoy" pod "409ba5bd-00c3-401d-bbda-e6191877887f" (UID: "409ba5bd-00c3-401d-bbda-e6191877887f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.824918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.824520    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-certs" pod "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" (UID: "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.824918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.824653    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-kube-api-access-tr9n6" pod "409ba5bd-00c3-401d-bbda-e6191877887f" (UID: "409ba5bd-00c3-401d-bbda-e6191877887f"). InnerVolumeSpecName "kube-api-access-tr9n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:45.824918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.824672    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-credential-socket" pod "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" (UID: "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.826088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.826052    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-podinfo" pod "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" (UID: "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:36:45.826416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.826390    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-token" pod "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" (UID: "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:45.826824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.826795    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-token" pod "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" (UID: "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:45.826917 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.826851    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-istio-token" pod "409ba5bd-00c3-401d-bbda-e6191877887f" (UID: "409ba5bd-00c3-401d-bbda-e6191877887f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:45.827428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.827396    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-certs" pod "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" (UID: "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.827561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.827534    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-podinfo" pod "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" (UID: "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:36:45.827680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.827656    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-kube-api-access-q4dw8" pod "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" (UID: "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c"). InnerVolumeSpecName "kube-api-access-q4dw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:45.828523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.827833    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-socket" pod "409ba5bd-00c3-401d-bbda-e6191877887f" (UID: "409ba5bd-00c3-401d-bbda-e6191877887f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.828523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.827886    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-socket" pod "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" (UID: "7b41774c-4f2c-4ba3-bae3-36b4e1b0150c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.828523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.828437    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-envoy" pod "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" (UID: "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.829575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.829555    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-kube-api-access-2h5pw" pod "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" (UID: "35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8"). InnerVolumeSpecName "kube-api-access-2h5pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:45.877480 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.877458    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:36:45.920283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920249    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b62919-1afd-4451-a026-4b69da048166-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b62919-1afd-4451-a026-4b69da048166-istiod-ca-cert\") pod \"c9b62919-1afd-4451-a026-4b69da048166\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") "
Nov 02 21:36:45.920464 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920293    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-kube-api-access-tgm46\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-kube-api-access-tgm46\") pod \"c9b62919-1afd-4451-a026-4b69da048166\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") "
Nov 02 21:36:45.920464 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920338    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-data\") pod \"c9b62919-1afd-4451-a026-4b69da048166\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") "
Nov 02 21:36:45.920464 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920370    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-envoy\") pod \"c9b62919-1afd-4451-a026-4b69da048166\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") "
Nov 02 21:36:45.920464 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920409    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9b62919-1afd-4451-a026-4b69da048166-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b62919-1afd-4451-a026-4b69da048166-istio-podinfo\") pod \"c9b62919-1afd-4451-a026-4b69da048166\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") "
Nov 02 21:36:45.920695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920486    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-istio-token\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-istio-token\") pod \"c9b62919-1afd-4451-a026-4b69da048166\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") "
Nov 02 21:36:45.920695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920537    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-socket\") pod \"c9b62919-1afd-4451-a026-4b69da048166\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") "
Nov 02 21:36:45.920695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920571    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-certs\") pod \"c9b62919-1afd-4451-a026-4b69da048166\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") "
Nov 02 21:36:45.920695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920605    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-credential-socket\") pod \"c9b62919-1afd-4451-a026-4b69da048166\" (UID: \"c9b62919-1afd-4451-a026-4b69da048166\") "
Nov 02 21:36:45.920907 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920690    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-data" pod "c9b62919-1afd-4451-a026-4b69da048166" (UID: "c9b62919-1afd-4451-a026-4b69da048166"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.920907 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920719    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b62919-1afd-4451-a026-4b69da048166-istiod-ca-cert" pod "c9b62919-1afd-4451-a026-4b69da048166" (UID: "c9b62919-1afd-4451-a026-4b69da048166"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:36:45.921032 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.920973    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-socket" pod "c9b62919-1afd-4451-a026-4b69da048166" (UID: "c9b62919-1afd-4451-a026-4b69da048166"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.921348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921254    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-credential-socket" pod "c9b62919-1afd-4451-a026-4b69da048166" (UID: "c9b62919-1afd-4451-a026-4b69da048166"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.921348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921280    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921303    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921317    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921330    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921343    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921357    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921369    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921381    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921393    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921408    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921421    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921434    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr9n6\" (UniqueName: \"kubernetes.io/projected/409ba5bd-00c3-401d-bbda-e6191877887f-kube-api-access-tr9n6\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921448    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921462    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921476    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409ba5bd-00c3-401d-bbda-e6191877887f-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921490    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4dw8\" (UniqueName: \"kubernetes.io/projected/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-kube-api-access-q4dw8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921503    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409ba5bd-00c3-401d-bbda-e6191877887f-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921520    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921534    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921550    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h5pw\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-kube-api-access-2h5pw\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921564    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921578    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9b62919-1afd-4451-a026-4b69da048166-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921592    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921608    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921622    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921636    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921650    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921664    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.921690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.921677    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:45.922522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.922498    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-certs" pod "c9b62919-1afd-4451-a026-4b69da048166" (UID: "c9b62919-1afd-4451-a026-4b69da048166"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:45.923137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.923097    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-kube-api-access-tgm46" pod "c9b62919-1afd-4451-a026-4b69da048166" (UID: "c9b62919-1afd-4451-a026-4b69da048166"). InnerVolumeSpecName "kube-api-access-tgm46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:45.923235 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.923217    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9b62919-1afd-4451-a026-4b69da048166-istio-podinfo" pod "c9b62919-1afd-4451-a026-4b69da048166" (UID: "c9b62919-1afd-4451-a026-4b69da048166"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:36:45.923708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.923692    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-istio-token" pod "c9b62919-1afd-4451-a026-4b69da048166" (UID: "c9b62919-1afd-4451-a026-4b69da048166"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:36:45.924371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:45.924351    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-envoy" pod "c9b62919-1afd-4451-a026-4b69da048166" (UID: "c9b62919-1afd-4451-a026-4b69da048166"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:36:46.023444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.023358    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgm46\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-kube-api-access-tgm46\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:46.023444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.023393    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:46.023444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.023407    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9b62919-1afd-4451-a026-4b69da048166-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:46.023444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.023419    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9b62919-1afd-4451-a026-4b69da048166-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:46.023444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.023431    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:46.023444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.023445    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9b62919-1afd-4451-a026-4b69da048166-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:36:46.099514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.099478    4528 generic.go:356] "Generic (PLEG): container finished" podID="409ba5bd-00c3-401d-bbda-e6191877887f" containerID="942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224" exitCode=0
Nov 02 21:36:46.099950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.099534    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"
Nov 02 21:36:46.099950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.099557    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" event={"ID":"409ba5bd-00c3-401d-bbda-e6191877887f","Type":"ContainerDied","Data":"942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224"}
Nov 02 21:36:46.099950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.099618    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk" event={"ID":"409ba5bd-00c3-401d-bbda-e6191877887f","Type":"ContainerDied","Data":"6eff04b7122963046fd0c7b9773082dcd7773b06322578297797c0037a83f83f"}
Nov 02 21:36:46.099950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.099647    4528 scope.go:122] "RemoveContainer" containerID="942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224"
Nov 02 21:36:46.101085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.101068    4528 generic.go:356] "Generic (PLEG): container finished" podID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerID="1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2" exitCode=0
Nov 02 21:36:46.101230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.101154    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"
Nov 02 21:36:46.101230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.101168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" event={"ID":"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8","Type":"ContainerDied","Data":"1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2"}
Nov 02 21:36:46.101230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.101222    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5" event={"ID":"35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8","Type":"ContainerDied","Data":"1e9eb0f921299c088ac5326a1ddfa0a40e74cca52d68b213a5d57923dbc2f885"}
Nov 02 21:36:46.103034 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.103006    4528 generic.go:356] "Generic (PLEG): container finished" podID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerID="fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6" exitCode=0
Nov 02 21:36:46.103196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.103059    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"
Nov 02 21:36:46.103196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.103084    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" event={"ID":"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c","Type":"ContainerDied","Data":"fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6"}
Nov 02 21:36:46.103196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.103155    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl" event={"ID":"7b41774c-4f2c-4ba3-bae3-36b4e1b0150c","Type":"ContainerDied","Data":"8d246825164134e68e53b1184301e47c20780987158156da6dc484e09ca512db"}
Nov 02 21:36:46.104493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.104472    4528 generic.go:356] "Generic (PLEG): container finished" podID="c9b62919-1afd-4451-a026-4b69da048166" containerID="6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a" exitCode=0
Nov 02 21:36:46.104606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.104511    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"
Nov 02 21:36:46.104606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.104531    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" event={"ID":"c9b62919-1afd-4451-a026-4b69da048166","Type":"ContainerDied","Data":"6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a"}
Nov 02 21:36:46.104606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.104566    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn" event={"ID":"c9b62919-1afd-4451-a026-4b69da048166","Type":"ContainerDied","Data":"8489ffe7b53431b618dab4daefb10fe24f94262c86fc69b9c31c4e6868702618"}
Nov 02 21:36:46.112164 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.112147    4528 scope.go:122] "RemoveContainer" containerID="942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224"
Nov 02 21:36:46.112486 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:46.112462    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224\": container with ID starting with 942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224 not found: ID does not exist" containerID="942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224"
Nov 02 21:36:46.112570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.112493    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224"} err="failed to get container status \"942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224\": rpc error: code = NotFound desc = could not find container \"942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224\": container with ID starting with 942544b57f06a13b0fb677150ef0f2568895a9fbf53320bbdef6cfdb1c0f9224 not found: ID does not exist"
Nov 02 21:36:46.112570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.112514    4528 scope.go:122] "RemoveContainer" containerID="1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2"
Nov 02 21:36:46.125170 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.125149    4528 scope.go:122] "RemoveContainer" containerID="1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2"
Nov 02 21:36:46.125464 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:46.125444    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2\": container with ID starting with 1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2 not found: ID does not exist" containerID="1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2"
Nov 02 21:36:46.125543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.125475    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2"} err="failed to get container status \"1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2\": rpc error: code = NotFound desc = could not find container \"1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2\": container with ID starting with 1256b3b1a33de94cd40806f7d0cc3153de391c49e0791e013702b7bd3926d7e2 not found: ID does not exist"
Nov 02 21:36:46.125543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.125500    4528 scope.go:122] "RemoveContainer" containerID="fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6"
Nov 02 21:36:46.134061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.134044    4528 scope.go:122] "RemoveContainer" containerID="f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f"
Nov 02 21:36:46.142173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.142152    4528 scope.go:122] "RemoveContainer" containerID="fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6"
Nov 02 21:36:46.142429 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:46.142412    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6\": container with ID starting with fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6 not found: ID does not exist" containerID="fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6"
Nov 02 21:36:46.142489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.142437    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6"} err="failed to get container status \"fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6\": rpc error: code = NotFound desc = could not find container \"fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6\": container with ID starting with fb1283b207243db072ff6fa393889a59a13fde61941ba02c6cc9c002655888d6 not found: ID does not exist"
Nov 02 21:36:46.142489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.142451    4528 scope.go:122] "RemoveContainer" containerID="f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f"
Nov 02 21:36:46.142733 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:46.142712    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f\": container with ID starting with f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f not found: ID does not exist" containerID="f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f"
Nov 02 21:36:46.142777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.142730    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f"} err="failed to get container status \"f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f\": rpc error: code = NotFound desc = could not find container \"f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f\": container with ID starting with f6436cd13463db05fe51aa96bf21bc964854081edc88a058c228226282a3a24f not found: ID does not exist"
Nov 02 21:36:46.142777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.142742    4528 scope.go:122] "RemoveContainer" containerID="6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a"
Nov 02 21:36:46.152232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.152217    4528 scope.go:122] "RemoveContainer" containerID="6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a"
Nov 02 21:36:46.152508 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:36:46.152491    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a\": container with ID starting with 6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a not found: ID does not exist" containerID="6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a"
Nov 02 21:36:46.152580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.152514    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a"} err="failed to get container status \"6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a\": rpc error: code = NotFound desc = could not find container \"6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a\": container with ID starting with 6128cfd1c36311743586ad4d8c69c631ed9b7c07a6fb731d74c4dd7f7455357a not found: ID does not exist"
Nov 02 21:36:46.407941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.407868    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"]
Nov 02 21:36:46.410378 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.410354    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-5rkzn"]
Nov 02 21:36:46.518304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.518276    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"]
Nov 02 21:36:46.522520 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.522491    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-vcjdl"]
Nov 02 21:36:46.630314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.630277    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"]
Nov 02 21:36:46.633772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.633747    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-cn8t5"]
Nov 02 21:36:46.743904 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.743800    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"]
Nov 02 21:36:46.746822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.746795    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-b2mgk"]
Nov 02 21:36:46.913434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.913398    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" path="/var/lib/kubelet/pods/35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8/volumes"
Nov 02 21:36:46.913877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.913864    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" path="/var/lib/kubelet/pods/409ba5bd-00c3-401d-bbda-e6191877887f/volumes"
Nov 02 21:36:46.914294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.914281    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" path="/var/lib/kubelet/pods/7b41774c-4f2c-4ba3-bae3-36b4e1b0150c/volumes"
Nov 02 21:36:46.914980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:46.914966    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b62919-1afd-4451-a026-4b69da048166" path="/var/lib/kubelet/pods/c9b62919-1afd-4451-a026-4b69da048166/volumes"
Nov 02 21:36:48.517339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517308    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"]
Nov 02 21:36:48.517861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517845    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy"
Nov 02 21:36:48.517902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517863    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy"
Nov 02 21:36:48.517902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517876    4528 cpu_manager.go:414] "containerMap: removing container" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy"
Nov 02 21:36:48.517902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517881    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy"
Nov 02 21:36:48.517902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517895    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy"
Nov 02 21:36:48.517902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517901    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517907    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517911    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517918    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517923    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517943    4528 cpu_manager.go:414] "containerMap: removing container" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517947    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517957    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.517961    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.518034    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.518044    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.518053    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bb0ce0-6b5f-444a-a9cc-93deccc6f8b8" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.518060    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b41774c-4f2c-4ba3-bae3-36b4e1b0150c" containerName="istio-proxy"
Nov 02 21:36:48.518066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.518067    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8bd196-f730-46cd-aa00-23352f984e61" containerName="istio-proxy"
Nov 02 21:36:48.518470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.518073    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b62919-1afd-4451-a026-4b69da048166" containerName="istio-proxy"
Nov 02 21:36:48.518470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.518081    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="409ba5bd-00c3-401d-bbda-e6191877887f" containerName="istio-proxy"
Nov 02 21:36:48.523336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.523315    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:36:48.560405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.560378    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skdn\" (UniqueName: \"kubernetes.io/projected/a78ce437-a092-4025-9da9-9069567c86d9-kube-api-access-8skdn\") pod \"payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l\" (UID: \"a78ce437-a092-4025-9da9-9069567c86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:36:48.560533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.560495    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78ce437-a092-4025-9da9-9069567c86d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l\" (UID: \"a78ce437-a092-4025-9da9-9069567c86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:36:48.575246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.575219    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"]
Nov 02 21:36:48.622092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.622057    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-15253e\""
Nov 02 21:36:48.663130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.662904    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skdn\" (UniqueName: \"kubernetes.io/projected/a78ce437-a092-4025-9da9-9069567c86d9-kube-api-access-8skdn\") pod \"payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l\" (UID: \"a78ce437-a092-4025-9da9-9069567c86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:36:48.663130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.663081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78ce437-a092-4025-9da9-9069567c86d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l\" (UID: \"a78ce437-a092-4025-9da9-9069567c86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:36:48.697167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.664145    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78ce437-a092-4025-9da9-9069567c86d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l\" (UID: \"a78ce437-a092-4025-9da9-9069567c86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:36:48.750142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.747183    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"]
Nov 02 21:36:48.754186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.752200    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:36:48.763134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.762677    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skdn\" (UniqueName: \"kubernetes.io/projected/a78ce437-a092-4025-9da9-9069567c86d9-kube-api-access-8skdn\") pod \"payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l\" (UID: \"a78ce437-a092-4025-9da9-9069567c86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:36:48.767136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.764619    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"]
Nov 02 21:36:48.805155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.804868    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-15253e-dockercfg-2vsql\""
Nov 02 21:36:48.836900 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.836873    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:36:48.865792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.865761    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgc7\" (UniqueName: \"kubernetes.io/projected/23866e02-cd31-4613-bd0d-8f7eb7ed6550-kube-api-access-vbgc7\") pod \"payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65\" (UID: \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:36:48.865946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.865805    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23866e02-cd31-4613-bd0d-8f7eb7ed6550-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65\" (UID: \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:36:48.967186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.967150    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgc7\" (UniqueName: \"kubernetes.io/projected/23866e02-cd31-4613-bd0d-8f7eb7ed6550-kube-api-access-vbgc7\") pod \"payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65\" (UID: \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:36:48.967186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.967191    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23866e02-cd31-4613-bd0d-8f7eb7ed6550-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65\" (UID: \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:36:48.968095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:48.968077    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23866e02-cd31-4613-bd0d-8f7eb7ed6550-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65\" (UID: \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:36:49.024295 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:49.024260    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgc7\" (UniqueName: \"kubernetes.io/projected/23866e02-cd31-4613-bd0d-8f7eb7ed6550-kube-api-access-vbgc7\") pod \"payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65\" (UID: \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:36:49.096856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:49.096790    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:36:49.206987 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:36:49.206955    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78ce437_a092_4025_9da9_9069567c86d9.slice/crio-93cfdc4ff538373a7dd6357a3c0de1e20479b3fa7fcec315ec40e209a1fa4eac WatchSource:0}: Error finding container 93cfdc4ff538373a7dd6357a3c0de1e20479b3fa7fcec315ec40e209a1fa4eac: Status 404 returned error can't find the container with id 93cfdc4ff538373a7dd6357a3c0de1e20479b3fa7fcec315ec40e209a1fa4eac
Nov 02 21:36:49.209697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:49.209671    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"]
Nov 02 21:36:49.464633 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:36:49.464596    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23866e02_cd31_4613_bd0d_8f7eb7ed6550.slice/crio-effa4fd556ded532607c3782970fe95a42f675df6c45614aa0785225ef34bcbf WatchSource:0}: Error finding container effa4fd556ded532607c3782970fe95a42f675df6c45614aa0785225ef34bcbf: Status 404 returned error can't find the container with id effa4fd556ded532607c3782970fe95a42f675df6c45614aa0785225ef34bcbf
Nov 02 21:36:49.468273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:49.468250    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"]
Nov 02 21:36:50.138189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:50.138146    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65" event={"ID":"23866e02-cd31-4613-bd0d-8f7eb7ed6550","Type":"ContainerStarted","Data":"7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362"}
Nov 02 21:36:50.138599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:50.138226    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:36:50.138599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:50.138247    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65" event={"ID":"23866e02-cd31-4613-bd0d-8f7eb7ed6550","Type":"ContainerStarted","Data":"effa4fd556ded532607c3782970fe95a42f675df6c45614aa0785225ef34bcbf"}
Nov 02 21:36:50.139658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:50.139633    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l" event={"ID":"a78ce437-a092-4025-9da9-9069567c86d9","Type":"ContainerStarted","Data":"995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2"}
Nov 02 21:36:50.139755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:50.139679    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:36:50.139755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:50.139695    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l" event={"ID":"a78ce437-a092-4025-9da9-9069567c86d9","Type":"ContainerStarted","Data":"93cfdc4ff538373a7dd6357a3c0de1e20479b3fa7fcec315ec40e209a1fa4eac"}
Nov 02 21:36:50.252851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:50.252801    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65" podStartSLOduration=2.252786738 podStartE2EDuration="2.252786738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:50.251050187 +0000 UTC m=+1697.514586509" watchObservedRunningTime="2026-11-02 21:36:50.252786738 +0000 UTC m=+1697.516323054"
Nov 02 21:36:57.048708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:57.048651    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l" podStartSLOduration=9.048638516 podStartE2EDuration="9.048638516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:50.385750558 +0000 UTC m=+1697.649286877" watchObservedRunningTime="2026-11-02 21:36:57.048638516 +0000 UTC m=+1704.312174826"
Nov 02 21:36:57.051424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:57.051396    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"]
Nov 02 21:36:57.101283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:57.101251    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380a24a3-6bvbg"]
Nov 02 21:36:58.913919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:36:58.913874    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a82826-97c5-4ba6-b32f-9d78606744a3" path="/var/lib/kubelet/pods/d7a82826-97c5-4ba6-b32f-9d78606744a3/volumes"
Nov 02 21:37:01.178848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:01.178814    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:37:01.179587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:01.179561    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:37:03.276454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:03.276418    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:37:14.113779 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:37:14.113736    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:37:14.324833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.324803    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"]
Nov 02 21:37:14.330296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.330271    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.343495 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.343468    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-token\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.343604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.343527    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrmm\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-kube-api-access-pqrmm\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.343604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.343586    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-credential-socket\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.343693 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.343661    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-podinfo\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.343728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.343709    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-data\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.343761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.343730    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206f69-8091-412f-9a80-e89cd60e50a5-istiod-ca-cert\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.343814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.343796    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-envoy\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.343852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.343825    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-certs\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.343852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.343845    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-socket\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.406687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.406600    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"]
Nov 02 21:37:14.436363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.436333    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7d3617-openshift-default-dockercfg-xd6pr\""
Nov 02 21:37:14.447513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.447470    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-podinfo\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.467388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.467355    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-data\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.467632 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.467612    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206f69-8091-412f-9a80-e89cd60e50a5-istiod-ca-cert\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.467796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.467781    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-envoy\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.467912 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.467897    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-certs\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.468026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.468011    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-socket\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.468324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.468303    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-token\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.468495 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.468480    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrmm\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-kube-api-access-pqrmm\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.468631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.468616    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-credential-socket\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.538392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.496003    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-socket\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.538682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.501501    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-certs\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.538992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.513495    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-data\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.539157 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.514339    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-credential-socket\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.539313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.536886    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206f69-8091-412f-9a80-e89cd60e50a5-istiod-ca-cert\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.542088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.542063    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-podinfo\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.544633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.544584    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-envoy\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.561229 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.561165    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-token\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.619347 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.619310    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrmm\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-kube-api-access-pqrmm\") pod \"e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") " pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:14.644200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:14.644166    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:15.079797 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:37:15.079767    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a206f69_8091_412f_9a80_e89cd60e50a5.slice/crio-1117088584f071a570ce729ed6b701110f5e81b57ecbe1e146975e8cbbdd5dab WatchSource:0}: Error finding container 1117088584f071a570ce729ed6b701110f5e81b57ecbe1e146975e8cbbdd5dab: Status 404 returned error can't find the container with id 1117088584f071a570ce729ed6b701110f5e81b57ecbe1e146975e8cbbdd5dab
Nov 02 21:37:15.082434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:15.082397    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:37:15.082527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:15.082460    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:37:15.082527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:15.082488    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:37:15.082908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:15.082885    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"]
Nov 02 21:37:15.339676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:15.339581    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" event={"ID":"7a206f69-8091-412f-9a80-e89cd60e50a5","Type":"ContainerStarted","Data":"75d747d1bf64e973afbb06870b021192b3ea0ae365b6b9f41fa18ef040a1a4b0"}
Nov 02 21:37:15.339676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:15.339638    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" event={"ID":"7a206f69-8091-412f-9a80-e89cd60e50a5","Type":"ContainerStarted","Data":"1117088584f071a570ce729ed6b701110f5e81b57ecbe1e146975e8cbbdd5dab"}
Nov 02 21:37:15.458839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:15.458785    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" podStartSLOduration=1.458771881 podStartE2EDuration="1.458771881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:15.456895183 +0000 UTC m=+1722.720431497" watchObservedRunningTime="2026-11-02 21:37:15.458771881 +0000 UTC m=+1722.722308198"
Nov 02 21:37:15.645027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:15.644928    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:16.094395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.094364    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"]
Nov 02 21:37:16.098333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.098313    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.157932 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.157904    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.158081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.157949    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.158081 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.158040    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.158199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.158096    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.158252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.158232    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.158330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.158313    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48771cb7-31e2-4192-99a2-e0e36f6d8737-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.158395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.158383    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.158461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.158441    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.158500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.158476    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kpks\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-kube-api-access-2kpks\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260301    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48771cb7-31e2-4192-99a2-e0e36f6d8737-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260383    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260424    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260447    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kpks\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-kube-api-access-2kpks\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260478    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260510    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260547    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260784    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260879 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260813    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260879 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260859    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260999 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260928    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.260999 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.260969    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.261121 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.261012    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.261272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.261248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48771cb7-31e2-4192-99a2-e0e36f6d8737-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.263064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.263041    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.263158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.263102    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.292412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.292385    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"]
Nov 02 21:37:16.361270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.359191    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kpks\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-kube-api-access-2kpks\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.384155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.383831    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.412976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.412931    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:16.650856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.650782    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:16.880645 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:37:16.880602    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48771cb7_31e2_4192_99a2_e0e36f6d8737.slice/crio-41f3bd0ca9d6a2fb9bb98b795580059e64dfd6dc176da2563bb4998d48481248 WatchSource:0}: Error finding container 41f3bd0ca9d6a2fb9bb98b795580059e64dfd6dc176da2563bb4998d48481248: Status 404 returned error can't find the container with id 41f3bd0ca9d6a2fb9bb98b795580059e64dfd6dc176da2563bb4998d48481248
Nov 02 21:37:16.883392 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.883371    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"]
Nov 02 21:37:16.883574 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.883545    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:37:16.883619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.883607    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:37:16.883658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:16.883635    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:37:17.398756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:17.398714    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" event={"ID":"48771cb7-31e2-4192-99a2-e0e36f6d8737","Type":"ContainerStarted","Data":"1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601"}
Nov 02 21:37:17.399282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:17.398777    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" event={"ID":"48771cb7-31e2-4192-99a2-e0e36f6d8737","Type":"ContainerStarted","Data":"41f3bd0ca9d6a2fb9bb98b795580059e64dfd6dc176da2563bb4998d48481248"}
Nov 02 21:37:17.399282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:17.399132    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:17.400059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:17.400041    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:37:17.424131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:17.415384    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 02 21:37:17.424131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:17.415480    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 02 21:37:17.424131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:17.415760    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:17.527722 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:17.527655    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" podStartSLOduration=1.5276345199999999 podStartE2EDuration="1.52763452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:17.52241954 +0000 UTC m=+1724.785955889" watchObservedRunningTime="2026-11-02 21:37:17.52763452 +0000 UTC m=+1724.791170839"
Nov 02 21:37:18.426555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:18.426521    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 02 21:37:18.427036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:18.426574    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 02 21:37:19.419133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:19.415497    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 02 21:37:19.419133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:19.415559    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 02 21:37:20.422134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:20.416471    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 02 21:37:20.422134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:20.416532    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 02 21:37:21.420134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:21.415456    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 02 21:37:21.420134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:21.415527    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 02 21:37:21.776651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:21.776611    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"]
Nov 02 21:37:21.782515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:21.782487    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:21.866133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:21.854927    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"]
Nov 02 21:37:21.936546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:21.921459    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7d3617\""
Nov 02 21:37:21.936546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:21.935879    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7d3617-dockercfg-g7m55\""
Nov 02 21:37:21.951138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:21.938511    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lht4\" (UniqueName: \"kubernetes.io/projected/790d52a0-4dcc-41d4-954d-691075117128-kube-api-access-9lht4\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx\" (UID: \"790d52a0-4dcc-41d4-954d-691075117128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:21.951138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:21.938556    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790d52a0-4dcc-41d4-954d-691075117128-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx\" (UID: \"790d52a0-4dcc-41d4-954d-691075117128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:22.079374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.042094    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lht4\" (UniqueName: \"kubernetes.io/projected/790d52a0-4dcc-41d4-954d-691075117128-kube-api-access-9lht4\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx\" (UID: \"790d52a0-4dcc-41d4-954d-691075117128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:22.079714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.079686    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790d52a0-4dcc-41d4-954d-691075117128-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx\" (UID: \"790d52a0-4dcc-41d4-954d-691075117128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:22.087478 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.087434    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"]
Nov 02 21:37:22.103912 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.103880    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790d52a0-4dcc-41d4-954d-691075117128-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx\" (UID: \"790d52a0-4dcc-41d4-954d-691075117128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:22.129134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.115626    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:22.149831 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.149764    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lht4\" (UniqueName: \"kubernetes.io/projected/790d52a0-4dcc-41d4-954d-691075117128-kube-api-access-9lht4\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx\" (UID: \"790d52a0-4dcc-41d4-954d-691075117128\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:22.152069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.152039    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"]
Nov 02 21:37:22.283188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.283151    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2\" (UID: \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:22.283359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.283330    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcld9\" (UniqueName: \"kubernetes.io/projected/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-kube-api-access-xcld9\") pod \"payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2\" (UID: \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:22.384783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.384700    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcld9\" (UniqueName: \"kubernetes.io/projected/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-kube-api-access-xcld9\") pod \"payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2\" (UID: \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:22.384962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.384815    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2\" (UID: \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:22.385535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.385516    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2\" (UID: \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:22.397666 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.397637    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:22.419534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.419498    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 02 21:37:22.420658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.420627    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 02 21:37:22.441666 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.441642    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcld9\" (UniqueName: \"kubernetes.io/projected/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-kube-api-access-xcld9\") pod \"payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2\" (UID: \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:22.471060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.471023    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:22.795389 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:37:22.795357    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790d52a0_4dcc_41d4_954d_691075117128.slice/crio-19f8579041713b47e8db1a1a904808a521f8c9a813be3d34cbc384a42f73501d WatchSource:0}: Error finding container 19f8579041713b47e8db1a1a904808a521f8c9a813be3d34cbc384a42f73501d: Status 404 returned error can't find the container with id 19f8579041713b47e8db1a1a904808a521f8c9a813be3d34cbc384a42f73501d
Nov 02 21:37:22.798377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.798354    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"]
Nov 02 21:37:22.891257 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:37:22.891229    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacfb6c5_09e6_4dfb_80cb_22336fdaac43.slice/crio-78c6aa43833124cf203c5f25231a1a9dc532c6414b162c05c1988a8172fbf124 WatchSource:0}: Error finding container 78c6aa43833124cf203c5f25231a1a9dc532c6414b162c05c1988a8172fbf124: Status 404 returned error can't find the container with id 78c6aa43833124cf203c5f25231a1a9dc532c6414b162c05c1988a8172fbf124
Nov 02 21:37:22.893018 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:22.892996    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"]
Nov 02 21:37:23.461179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.461141    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2" event={"ID":"bacfb6c5-09e6-4dfb-80cb-22336fdaac43","Type":"ContainerStarted","Data":"36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f"}
Nov 02 21:37:23.461383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.461204    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2" event={"ID":"bacfb6c5-09e6-4dfb-80cb-22336fdaac43","Type":"ContainerStarted","Data":"78c6aa43833124cf203c5f25231a1a9dc532c6414b162c05c1988a8172fbf124"}
Nov 02 21:37:23.461462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.461446    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:23.462926 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.462896    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx" event={"ID":"790d52a0-4dcc-41d4-954d-691075117128","Type":"ContainerStarted","Data":"fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869"}
Nov 02 21:37:23.463055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.462950    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx" event={"ID":"790d52a0-4dcc-41d4-954d-691075117128","Type":"ContainerStarted","Data":"19f8579041713b47e8db1a1a904808a521f8c9a813be3d34cbc384a42f73501d"}
Nov 02 21:37:23.463055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.462982    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:23.579042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.578990    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2" podStartSLOduration=2.57897484 podStartE2EDuration="2.57897484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:23.574242079 +0000 UTC m=+1730.837778412" watchObservedRunningTime="2026-11-02 21:37:23.57897484 +0000 UTC m=+1730.842511156"
Nov 02 21:37:23.730830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.730708    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx" podStartSLOduration=2.730685368 podStartE2EDuration="2.730685368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:23.69315365 +0000 UTC m=+1730.956689970" watchObservedRunningTime="2026-11-02 21:37:23.730685368 +0000 UTC m=+1730.994221686"
Nov 02 21:37:23.860104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.860074    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:23.862373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.862346    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:23.865035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:23.865011    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:37:29.606509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:29.606472    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7471-3484f5"]
Nov 02 21:37:29.610499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:29.610481    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-3484f5"
Nov 02 21:37:29.659533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:29.659501    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7471-3484f5"]
Nov 02 21:37:29.705657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:29.705630    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5v2nf\""
Nov 02 21:37:29.710434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:29.710411    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t82ts\" (UniqueName: \"kubernetes.io/projected/6271d098-7478-43e2-81ec-1f33de399d2d-kube-api-access-t82ts\") pod \"test-curl-7471-3484f5\" (UID: \"6271d098-7478-43e2-81ec-1f33de399d2d\") " pod="opendatahub/test-curl-7471-3484f5"
Nov 02 21:37:29.811578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:29.811545    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t82ts\" (UniqueName: \"kubernetes.io/projected/6271d098-7478-43e2-81ec-1f33de399d2d-kube-api-access-t82ts\") pod \"test-curl-7471-3484f5\" (UID: \"6271d098-7478-43e2-81ec-1f33de399d2d\") " pod="opendatahub/test-curl-7471-3484f5"
Nov 02 21:37:29.868577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:29.868505    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t82ts\" (UniqueName: \"kubernetes.io/projected/6271d098-7478-43e2-81ec-1f33de399d2d-kube-api-access-t82ts\") pod \"test-curl-7471-3484f5\" (UID: \"6271d098-7478-43e2-81ec-1f33de399d2d\") " pod="opendatahub/test-curl-7471-3484f5"
Nov 02 21:37:29.923494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:29.923468    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-3484f5"
Nov 02 21:37:30.293250 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:37:30.293218    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6271d098_7478_43e2_81ec_1f33de399d2d.slice/crio-bf15bb1286946cc6111f4e4096a09a6dae84f5d90920bfd09f29bbc5c71fa0a7 WatchSource:0}: Error finding container bf15bb1286946cc6111f4e4096a09a6dae84f5d90920bfd09f29bbc5c71fa0a7: Status 404 returned error can't find the container with id bf15bb1286946cc6111f4e4096a09a6dae84f5d90920bfd09f29bbc5c71fa0a7
Nov 02 21:37:30.295354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:30.295338    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:37:30.295872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:30.295849    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7471-3484f5"]
Nov 02 21:37:30.504350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:30.504312    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7471-3484f5" event={"ID":"6271d098-7478-43e2-81ec-1f33de399d2d","Type":"ContainerStarted","Data":"bf15bb1286946cc6111f4e4096a09a6dae84f5d90920bfd09f29bbc5c71fa0a7"}
Nov 02 21:37:31.089367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.089336    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"]
Nov 02 21:37:31.094541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.094524    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.229961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.229927    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z596c\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-kube-api-access-z596c\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.230165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.229978    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.230165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.230054    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.230165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.230095    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.230287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.230217    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.230287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.230254    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe5a031-de99-4fb9-b118-68ba39cf459b-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.230378 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.230356    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.230458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.230444    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.230492 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.230471    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.317273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.317240    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"]
Nov 02 21:37:31.331680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.331641    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.331839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.331741    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.331839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.331769    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.331839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.331831    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z596c\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-kube-api-access-z596c\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.331885    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.331929    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.331967    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.332038    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.332101    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe5a031-de99-4fb9-b118-68ba39cf459b-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.332167    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.332202    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.332256    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.332459    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.332813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.332791    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe5a031-de99-4fb9-b118-68ba39cf459b-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.334641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.334611    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.334965 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.334942    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.389186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.389102    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.389305 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.389181    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z596c\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-kube-api-access-z596c\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.409192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.409155    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:31.832526 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:37:31.832491    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe5a031_de99_4fb9_b118_68ba39cf459b.slice/crio-2ba637b610e12b6d1addb6ad2e145dd646ea7c10cdf79dcf8558d4fd3e8cc468 WatchSource:0}: Error finding container 2ba637b610e12b6d1addb6ad2e145dd646ea7c10cdf79dcf8558d4fd3e8cc468: Status 404 returned error can't find the container with id 2ba637b610e12b6d1addb6ad2e145dd646ea7c10cdf79dcf8558d4fd3e8cc468
Nov 02 21:37:31.835050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.835026    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"]
Nov 02 21:37:31.835310 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.835279    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:37:31.835370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.835355    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:37:31.835411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:31.835396    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:37:32.522249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:32.522200    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" event={"ID":"ebe5a031-de99-4fb9-b118-68ba39cf459b","Type":"ContainerStarted","Data":"ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7"}
Nov 02 21:37:32.522753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:32.522267    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" event={"ID":"ebe5a031-de99-4fb9-b118-68ba39cf459b","Type":"ContainerStarted","Data":"2ba637b610e12b6d1addb6ad2e145dd646ea7c10cdf79dcf8558d4fd3e8cc468"}
Nov 02 21:37:32.767868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:32.767813    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" podStartSLOduration=1.7677957420000001 podStartE2EDuration="1.767795742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:32.716708432 +0000 UTC m=+1739.980244761" watchObservedRunningTime="2026-11-02 21:37:32.767795742 +0000 UTC m=+1740.031332059"
Nov 02 21:37:33.413583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:33.410495    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:33.426418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:33.426369    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 21:37:33.426594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:33.426462    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 21:37:33.534474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:33.534439    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7471-3484f5" event={"ID":"6271d098-7478-43e2-81ec-1f33de399d2d","Type":"ContainerStarted","Data":"a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9"}
Nov 02 21:37:33.647301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:33.647183    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7471-3484f5" podStartSLOduration=1.578519081 podStartE2EDuration="4.647165404s" totalImagesPullingTime="3.068646323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:33.645884035 +0000 UTC m=+1740.909420355" watchObservedRunningTime="2026-11-02 21:37:33.647165404 +0000 UTC m=+1740.910701722"
Nov 02 21:37:34.045624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:34.045589    4528 scope.go:122] "RemoveContainer" containerID="7342e82de7777cee60150b8c925a2b98fd04b77562419dc608d8990d991a474c"
Nov 02 21:37:34.421188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:34.412460    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 21:37:34.421188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:34.412522    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 21:37:34.472386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:34.472357    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:37:34.472553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:34.472452    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:37:35.074997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:35.074967    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7471-3484f5"]
Nov 02 21:37:35.077946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:35.077918    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7471-3484f5"]
Nov 02 21:37:35.426285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:35.426189    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 21:37:35.426285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:35.426252    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 21:37:35.546783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:35.546728    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7471-3484f5" podUID="6271d098-7478-43e2-81ec-1f33de399d2d" containerName="test-curl-7471-3484f5" containerID="cri-o://a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9" gracePeriod=2
Nov 02 21:37:35.599222 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:37:35.599183    4528 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-3484f5\" is forbidden: User \"system:node:ip-10-0-133-86.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-86.us-west-2.compute.internal' and this object" podUID="6271d098-7478-43e2-81ec-1f33de399d2d" pod="opendatahub/test-curl-7471-3484f5"
Nov 02 21:37:36.420570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:36.420530    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 21:37:36.421242 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:36.421206    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 21:37:37.421602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:37.421560    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 21:37:37.422243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:37.422210    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 21:37:37.847294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:37.847269    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-3484f5"
Nov 02 21:37:37.952662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:37.952631    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6271d098-7478-43e2-81ec-1f33de399d2d-kube-api-access-t82ts\" (UniqueName: \"kubernetes.io/projected/6271d098-7478-43e2-81ec-1f33de399d2d-kube-api-access-t82ts\") pod \"6271d098-7478-43e2-81ec-1f33de399d2d\" (UID: \"6271d098-7478-43e2-81ec-1f33de399d2d\") "
Nov 02 21:37:37.954798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:37.954771    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6271d098-7478-43e2-81ec-1f33de399d2d-kube-api-access-t82ts" pod "6271d098-7478-43e2-81ec-1f33de399d2d" (UID: "6271d098-7478-43e2-81ec-1f33de399d2d"). InnerVolumeSpecName "kube-api-access-t82ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:37:38.054843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.054806    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t82ts\" (UniqueName: \"kubernetes.io/projected/6271d098-7478-43e2-81ec-1f33de399d2d-kube-api-access-t82ts\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:37:38.498876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.498752    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:38.501264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.501243    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:38.502089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.502067    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:37:38.566171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.566137    4528 generic.go:356] "Generic (PLEG): container finished" podID="6271d098-7478-43e2-81ec-1f33de399d2d" containerID="a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9" exitCode=137
Nov 02 21:37:38.566352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.566202    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-3484f5"
Nov 02 21:37:38.566352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.566241    4528 scope.go:122] "RemoveContainer" containerID="a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9"
Nov 02 21:37:38.577749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.577726    4528 scope.go:122] "RemoveContainer" containerID="a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9"
Nov 02 21:37:38.578042 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:37:38.578025    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9\": container with ID starting with a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9 not found: ID does not exist" containerID="a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9"
Nov 02 21:37:38.578083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.578052    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9"} err="failed to get container status \"a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9\": rpc error: code = NotFound desc = could not find container \"a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9\": container with ID starting with a1c9114d7d100cc2b39f9b44e2a3047fc7a4fdf206bdae69478d3c4ea0fde7e9 not found: ID does not exist"
Nov 02 21:37:38.741795 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:37:38.741746    4528 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-3484f5\" is forbidden: User \"system:node:ip-10-0-133-86.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-86.us-west-2.compute.internal' and this object" podUID="6271d098-7478-43e2-81ec-1f33de399d2d" pod="opendatahub/test-curl-7471-3484f5"
Nov 02 21:37:38.913742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:38.913698    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6271d098-7478-43e2-81ec-1f33de399d2d" path="/var/lib/kubelet/pods/6271d098-7478-43e2-81ec-1f33de399d2d/volumes"
Nov 02 21:37:41.516530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:41.516497    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"]
Nov 02 21:37:44.113176 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:37:44.113132    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:37:46.516956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:46.516906    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46" podUID="aa666493-4f89-4a72-8a3e-ffc854458af3" containerName="payload-processing" containerID="cri-o://f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0" gracePeriod=25
Nov 02 21:37:46.801291 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:46.801266    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:37:46.900470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:46.900435    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa666493-4f89-4a72-8a3e-ffc854458af3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa666493-4f89-4a72-8a3e-ffc854458af3-plugins-config-volume\") pod \"aa666493-4f89-4a72-8a3e-ffc854458af3\" (UID: \"aa666493-4f89-4a72-8a3e-ffc854458af3\") "
Nov 02 21:37:46.900633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:46.900500    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa666493-4f89-4a72-8a3e-ffc854458af3-kube-api-access-f86w8\" (UniqueName: \"kubernetes.io/projected/aa666493-4f89-4a72-8a3e-ffc854458af3-kube-api-access-f86w8\") pod \"aa666493-4f89-4a72-8a3e-ffc854458af3\" (UID: \"aa666493-4f89-4a72-8a3e-ffc854458af3\") "
Nov 02 21:37:46.900783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:46.900762    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa666493-4f89-4a72-8a3e-ffc854458af3-plugins-config-volume" pod "aa666493-4f89-4a72-8a3e-ffc854458af3" (UID: "aa666493-4f89-4a72-8a3e-ffc854458af3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:37:46.900981 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:46.900941    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa666493-4f89-4a72-8a3e-ffc854458af3-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:37:46.902646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:46.902618    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa666493-4f89-4a72-8a3e-ffc854458af3-kube-api-access-f86w8" pod "aa666493-4f89-4a72-8a3e-ffc854458af3" (UID: "aa666493-4f89-4a72-8a3e-ffc854458af3"). InnerVolumeSpecName "kube-api-access-f86w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:37:47.002769 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.002726    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f86w8\" (UniqueName: \"kubernetes.io/projected/aa666493-4f89-4a72-8a3e-ffc854458af3-kube-api-access-f86w8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:37:47.621831 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.621793    4528 generic.go:356] "Generic (PLEG): container finished" podID="aa666493-4f89-4a72-8a3e-ffc854458af3" containerID="f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0" exitCode=0
Nov 02 21:37:47.622325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.621835    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"
Nov 02 21:37:47.622325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.621873    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46" event={"ID":"aa666493-4f89-4a72-8a3e-ffc854458af3","Type":"ContainerDied","Data":"f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0"}
Nov 02 21:37:47.622325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.621923    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46" event={"ID":"aa666493-4f89-4a72-8a3e-ffc854458af3","Type":"ContainerDied","Data":"131d57655185b8a648ef860f009adce9ddca5b4803729ba94196b426e8a7f9f5"}
Nov 02 21:37:47.622325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.621945    4528 scope.go:122] "RemoveContainer" containerID="f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0"
Nov 02 21:37:47.632322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.632297    4528 scope.go:122] "RemoveContainer" containerID="f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0"
Nov 02 21:37:47.632585 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:37:47.632560    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0\": container with ID starting with f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0 not found: ID does not exist" containerID="f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0"
Nov 02 21:37:47.632646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.632592    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0"} err="failed to get container status \"f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0\": rpc error: code = NotFound desc = could not find container \"f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0\": container with ID starting with f07344564af0e6a373dc0610b9b2d71860465bf99193d01ce258688ee11f2eb0 not found: ID does not exist"
Nov 02 21:37:47.787281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.787245    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"]
Nov 02 21:37:47.837023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:47.836994    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-682116-7c5c94c747-6vr46"]
Nov 02 21:37:48.913582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:48.913548    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa666493-4f89-4a72-8a3e-ffc854458af3" path="/var/lib/kubelet/pods/aa666493-4f89-4a72-8a3e-ffc854458af3/volumes"
Nov 02 21:37:54.457943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:37:54.457909    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:38:03.327137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:03.327090    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"]
Nov 02 21:38:03.327610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:03.327333    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy" containerID="cri-o://2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc" gracePeriod=30
Nov 02 21:38:03.379196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:03.379165    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"]
Nov 02 21:38:03.379383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:03.379362    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy" containerID="cri-o://21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b" gracePeriod=30
Nov 02 21:38:03.385138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:03.384531    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"]
Nov 02 21:38:03.385138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:03.384894    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy" containerID="cri-o://42e28435607b4942a4aec95ce65eccc6886d10510f3bb4f91a8628bd9dad4ac6" gracePeriod=30
Nov 02 21:38:03.429653 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:03.429616    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"]
Nov 02 21:38:03.429912 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:03.429861    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy" containerID="cri-o://ab0b54b75601731a7378fecdd44b2d116881189288c8623f67428a760de76e98" gracePeriod=30
Nov 02 21:38:05.808620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:05.808533    4528 generic.go:356] "Generic (PLEG): container finished" podID="f971579a-0d19-4119-9924-df6165cfb992" containerID="e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898" exitCode=0
Nov 02 21:38:05.809400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:05.809370    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" event={"ID":"f971579a-0d19-4119-9924-df6165cfb992","Type":"ContainerDied","Data":"e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898"}
Nov 02 21:38:05.810145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:05.810125    4528 scope.go:122] "RemoveContainer" containerID="e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898"
Nov 02 21:38:05.829195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:05.829150    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:38:05.830687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:05.830658    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:38:05.830885 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:05.830866    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:38:06.832416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:06.832378    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" event={"ID":"f971579a-0d19-4119-9924-df6165cfb992","Type":"ContainerStarted","Data":"7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df"}
Nov 02 21:38:07.774450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:07.774418    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:38:07.774636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:07.774617    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 21:38:07.774693 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:07.774674    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 21:38:08.745864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.744881    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:38:08.775880 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.775822    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 21:38:08.776791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.776224    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 21:38:08.799882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.799855    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:38:08.846920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.846887    4528 generic.go:356] "Generic (PLEG): container finished" podID="771361df-3dd7-4787-bf88-53a214521356" containerID="2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc" exitCode=0
Nov 02 21:38:08.847201 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.846987    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" event={"ID":"771361df-3dd7-4787-bf88-53a214521356","Type":"ContainerDied","Data":"2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc"}
Nov 02 21:38:08.847344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.847019    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"
Nov 02 21:38:08.847344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.847233    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9" event={"ID":"771361df-3dd7-4787-bf88-53a214521356","Type":"ContainerDied","Data":"b3b7d02f7880853ec52c19f213968b1a52d3dcaffe5247d6eb9b6f3751147801"}
Nov 02 21:38:08.847344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.847275    4528 scope.go:122] "RemoveContainer" containerID="2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc"
Nov 02 21:38:08.849011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.848904    4528 generic.go:356] "Generic (PLEG): container finished" podID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerID="21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b" exitCode=0
Nov 02 21:38:08.849011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.848986    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"
Nov 02 21:38:08.849160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.849027    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" event={"ID":"b32c3f43-49f5-412b-b226-f1d0ce064ac7","Type":"ContainerDied","Data":"21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b"}
Nov 02 21:38:08.849160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.849071    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk" event={"ID":"b32c3f43-49f5-412b-b226-f1d0ce064ac7","Type":"ContainerDied","Data":"99301da0e2eca4afa5bd3074c405005efa6a9574016086957f5844e5eb21889c"}
Nov 02 21:38:08.850511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.850494    4528 generic.go:356] "Generic (PLEG): container finished" podID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerID="ab0b54b75601731a7378fecdd44b2d116881189288c8623f67428a760de76e98" exitCode=0
Nov 02 21:38:08.850612 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.850566    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" event={"ID":"0d74ae5c-80fe-434b-9313-e42f60ae0518","Type":"ContainerDied","Data":"ab0b54b75601731a7378fecdd44b2d116881189288c8623f67428a760de76e98"}
Nov 02 21:38:08.851864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.851802    4528 generic.go:356] "Generic (PLEG): container finished" podID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerID="42e28435607b4942a4aec95ce65eccc6886d10510f3bb4f91a8628bd9dad4ac6" exitCode=0
Nov 02 21:38:08.851864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.851830    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" event={"ID":"22e12023-6b7c-44a9-b1d1-016fd7e1c43b","Type":"ContainerDied","Data":"42e28435607b4942a4aec95ce65eccc6886d10510f3bb4f91a8628bd9dad4ac6"}
Nov 02 21:38:08.853210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853187    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-data\") pod \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") "
Nov 02 21:38:08.853290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853233    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-certs\") pod \"771361df-3dd7-4787-bf88-53a214521356\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") "
Nov 02 21:38:08.853290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853281    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-token\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-token\") pod \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") "
Nov 02 21:38:08.853403 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853348    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-data\") pod \"771361df-3dd7-4787-bf88-53a214521356\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") "
Nov 02 21:38:08.853403 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853378    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-istio-token\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-istio-token\") pod \"771361df-3dd7-4787-bf88-53a214521356\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") "
Nov 02 21:38:08.853518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853423    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-socket\") pod \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") "
Nov 02 21:38:08.853518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853428    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-data" pod "b32c3f43-49f5-412b-b226-f1d0ce064ac7" (UID: "b32c3f43-49f5-412b-b226-f1d0ce064ac7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.853518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853495    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-envoy\") pod \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") "
Nov 02 21:38:08.853676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853534    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-credential-socket\") pod \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") "
Nov 02 21:38:08.853676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853553    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771361df-3dd7-4787-bf88-53a214521356-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771361df-3dd7-4787-bf88-53a214521356-istiod-ca-cert\") pod \"771361df-3dd7-4787-bf88-53a214521356\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") "
Nov 02 21:38:08.853676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853600    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istiod-ca-cert\") pod \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") "
Nov 02 21:38:08.853676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853618    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-certs\") pod \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") "
Nov 02 21:38:08.853676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853655    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-certs" pod "771361df-3dd7-4787-bf88-53a214521356" (UID: "771361df-3dd7-4787-bf88-53a214521356"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.853939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853659    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-data" pod "771361df-3dd7-4787-bf88-53a214521356" (UID: "771361df-3dd7-4787-bf88-53a214521356"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.853939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853684    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-credential-socket\") pod \"771361df-3dd7-4787-bf88-53a214521356\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") "
Nov 02 21:38:08.853939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853704    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-socket\") pod \"771361df-3dd7-4787-bf88-53a214521356\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") "
Nov 02 21:38:08.853939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853721    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-kube-api-access-447bq\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-kube-api-access-447bq\") pod \"771361df-3dd7-4787-bf88-53a214521356\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") "
Nov 02 21:38:08.853939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853760    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-podinfo\") pod \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") "
Nov 02 21:38:08.853939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853822    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-kube-api-access-4vx2z\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-kube-api-access-4vx2z\") pod \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\" (UID: \"b32c3f43-49f5-412b-b226-f1d0ce064ac7\") "
Nov 02 21:38:08.853939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853867    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-envoy\") pod \"771361df-3dd7-4787-bf88-53a214521356\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") "
Nov 02 21:38:08.853939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853879    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-credential-socket" pod "b32c3f43-49f5-412b-b226-f1d0ce064ac7" (UID: "b32c3f43-49f5-412b-b226-f1d0ce064ac7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.853939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.853937    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/771361df-3dd7-4787-bf88-53a214521356-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771361df-3dd7-4787-bf88-53a214521356-istio-podinfo\") pod \"771361df-3dd7-4787-bf88-53a214521356\" (UID: \"771361df-3dd7-4787-bf88-53a214521356\") "
Nov 02 21:38:08.854364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.854156    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-socket" pod "b32c3f43-49f5-412b-b226-f1d0ce064ac7" (UID: "b32c3f43-49f5-412b-b226-f1d0ce064ac7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.854633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.854610    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.854727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.854640    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.854727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.854657    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.854727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.854672    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.854727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.854686    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.854914 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.854811    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-socket" pod "771361df-3dd7-4787-bf88-53a214521356" (UID: "771361df-3dd7-4787-bf88-53a214521356"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.855833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.855061    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771361df-3dd7-4787-bf88-53a214521356-istiod-ca-cert" pod "771361df-3dd7-4787-bf88-53a214521356" (UID: "771361df-3dd7-4787-bf88-53a214521356"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:38:08.855833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.855332    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istiod-ca-cert" pod "b32c3f43-49f5-412b-b226-f1d0ce064ac7" (UID: "b32c3f43-49f5-412b-b226-f1d0ce064ac7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:38:08.855833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.855507    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-certs" pod "b32c3f43-49f5-412b-b226-f1d0ce064ac7" (UID: "b32c3f43-49f5-412b-b226-f1d0ce064ac7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.855833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.855628    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-credential-socket" pod "771361df-3dd7-4787-bf88-53a214521356" (UID: "771361df-3dd7-4787-bf88-53a214521356"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.856399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.856259    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-token" pod "b32c3f43-49f5-412b-b226-f1d0ce064ac7" (UID: "b32c3f43-49f5-412b-b226-f1d0ce064ac7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:08.856705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.856681    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-istio-token" pod "771361df-3dd7-4787-bf88-53a214521356" (UID: "771361df-3dd7-4787-bf88-53a214521356"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:08.858556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.858533    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-envoy" pod "b32c3f43-49f5-412b-b226-f1d0ce064ac7" (UID: "b32c3f43-49f5-412b-b226-f1d0ce064ac7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.858641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.858587    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/771361df-3dd7-4787-bf88-53a214521356-istio-podinfo" pod "771361df-3dd7-4787-bf88-53a214521356" (UID: "771361df-3dd7-4787-bf88-53a214521356"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:38:08.858709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.858677    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-envoy" pod "771361df-3dd7-4787-bf88-53a214521356" (UID: "771361df-3dd7-4787-bf88-53a214521356"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.858709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.858692    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-podinfo" pod "b32c3f43-49f5-412b-b226-f1d0ce064ac7" (UID: "b32c3f43-49f5-412b-b226-f1d0ce064ac7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:38:08.859265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.859238    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-kube-api-access-4vx2z" pod "b32c3f43-49f5-412b-b226-f1d0ce064ac7" (UID: "b32c3f43-49f5-412b-b226-f1d0ce064ac7"). InnerVolumeSpecName "kube-api-access-4vx2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:08.860027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.859994    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-kube-api-access-447bq" pod "771361df-3dd7-4787-bf88-53a214521356" (UID: "771361df-3dd7-4787-bf88-53a214521356"). InnerVolumeSpecName "kube-api-access-447bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:08.860689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.860664    4528 scope.go:122] "RemoveContainer" containerID="2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc"
Nov 02 21:38:08.861122 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:38:08.861081    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc\": container with ID starting with 2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc not found: ID does not exist" containerID="2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc"
Nov 02 21:38:08.861225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.861135    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc"} err="failed to get container status \"2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc\": rpc error: code = NotFound desc = could not find container \"2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc\": container with ID starting with 2756fa151be95ff9f8c895165326633787031595992ea39522e66602031b8bfc not found: ID does not exist"
Nov 02 21:38:08.861225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.861157    4528 scope.go:122] "RemoveContainer" containerID="21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b"
Nov 02 21:38:08.861225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.861094    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:38:08.864954 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.864938    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:38:08.869561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.869545    4528 scope.go:122] "RemoveContainer" containerID="21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b"
Nov 02 21:38:08.869793 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:38:08.869777    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b\": container with ID starting with 21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b not found: ID does not exist" containerID="21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b"
Nov 02 21:38:08.869857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.869798    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b"} err="failed to get container status \"21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b\": rpc error: code = NotFound desc = could not find container \"21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b\": container with ID starting with 21d23eee935cefa7c12ad66c9e143a02237eebb258d7d07d5851f9375af87f2b not found: ID does not exist"
Nov 02 21:38:08.955835 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.955813    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-token\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-token\") pod \"0d74ae5c-80fe-434b-9313-e42f60ae0518\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") "
Nov 02 21:38:08.955949 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.955856    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-credential-socket\") pod \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") "
Nov 02 21:38:08.955949 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.955901    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-podinfo\") pod \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") "
Nov 02 21:38:08.956073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956000    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d74ae5c-80fe-434b-9313-e42f60ae0518-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74ae5c-80fe-434b-9313-e42f60ae0518-istiod-ca-cert\") pod \"0d74ae5c-80fe-434b-9313-e42f60ae0518\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") "
Nov 02 21:38:08.956073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956057    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-socket\") pod \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") "
Nov 02 21:38:08.956198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956085    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-certs\") pod \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") "
Nov 02 21:38:08.956198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956134    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-data\") pod \"0d74ae5c-80fe-434b-9313-e42f60ae0518\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") "
Nov 02 21:38:08.956312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956208    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-token\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-token\") pod \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") "
Nov 02 21:38:08.956312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956253    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-socket\") pod \"0d74ae5c-80fe-434b-9313-e42f60ae0518\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") "
Nov 02 21:38:08.956312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956304    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-data\") pod \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") "
Nov 02 21:38:08.956472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956346    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-envoy\") pod \"0d74ae5c-80fe-434b-9313-e42f60ae0518\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") "
Nov 02 21:38:08.956472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956379    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-socket" pod "22e12023-6b7c-44a9-b1d1-016fd7e1c43b" (UID: "22e12023-6b7c-44a9-b1d1-016fd7e1c43b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.956472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956400    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-podinfo\") pod \"0d74ae5c-80fe-434b-9313-e42f60ae0518\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") "
Nov 02 21:38:08.956644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956480    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-kube-api-access-2s2bv\") pod \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") "
Nov 02 21:38:08.956644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956531    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-certs\") pod \"0d74ae5c-80fe-434b-9313-e42f60ae0518\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") "
Nov 02 21:38:08.956644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956577    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istiod-ca-cert\") pod \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") "
Nov 02 21:38:08.956644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956602    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-credential-socket\") pod \"0d74ae5c-80fe-434b-9313-e42f60ae0518\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") "
Nov 02 21:38:08.956849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956665    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d74ae5c-80fe-434b-9313-e42f60ae0518-istiod-ca-cert" pod "0d74ae5c-80fe-434b-9313-e42f60ae0518" (UID: "0d74ae5c-80fe-434b-9313-e42f60ae0518"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:38:08.956849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956689    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-kube-api-access-rx9pb\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-kube-api-access-rx9pb\") pod \"0d74ae5c-80fe-434b-9313-e42f60ae0518\" (UID: \"0d74ae5c-80fe-434b-9313-e42f60ae0518\") "
Nov 02 21:38:08.956849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.956768    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-envoy\") pod \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\" (UID: \"22e12023-6b7c-44a9-b1d1-016fd7e1c43b\") "
Nov 02 21:38:08.957103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957076    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-credential-socket" pod "22e12023-6b7c-44a9-b1d1-016fd7e1c43b" (UID: "22e12023-6b7c-44a9-b1d1-016fd7e1c43b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.957459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957442    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957465    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-447bq\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-kube-api-access-447bq\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957481    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957494    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74ae5c-80fe-434b-9313-e42f60ae0518-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957508    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957523    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vx2z\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-kube-api-access-4vx2z\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957537    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957552    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771361df-3dd7-4787-bf88-53a214521356-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957567    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957581    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771361df-3dd7-4787-bf88-53a214521356-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957595    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957609    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771361df-3dd7-4787-bf88-53a214521356-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957624    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c3f43-49f5-412b-b226-f1d0ce064ac7-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957640    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c3f43-49f5-412b-b226-f1d0ce064ac7-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957654    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.957872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.957668    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771361df-3dd7-4787-bf88-53a214521356-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:08.959690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.959458    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-certs" pod "0d74ae5c-80fe-434b-9313-e42f60ae0518" (UID: "0d74ae5c-80fe-434b-9313-e42f60ae0518"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.959793 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.959689    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istiod-ca-cert" pod "22e12023-6b7c-44a9-b1d1-016fd7e1c43b" (UID: "22e12023-6b7c-44a9-b1d1-016fd7e1c43b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:38:08.959950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.959928    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-certs" pod "22e12023-6b7c-44a9-b1d1-016fd7e1c43b" (UID: "22e12023-6b7c-44a9-b1d1-016fd7e1c43b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.960037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.959944    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-credential-socket" pod "0d74ae5c-80fe-434b-9313-e42f60ae0518" (UID: "0d74ae5c-80fe-434b-9313-e42f60ae0518"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.960180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.960142    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-socket" pod "0d74ae5c-80fe-434b-9313-e42f60ae0518" (UID: "0d74ae5c-80fe-434b-9313-e42f60ae0518"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.960277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.960173    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-data" pod "22e12023-6b7c-44a9-b1d1-016fd7e1c43b" (UID: "22e12023-6b7c-44a9-b1d1-016fd7e1c43b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.960352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.960329    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-data" pod "0d74ae5c-80fe-434b-9313-e42f60ae0518" (UID: "0d74ae5c-80fe-434b-9313-e42f60ae0518"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.973466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.973442    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-token" pod "0d74ae5c-80fe-434b-9313-e42f60ae0518" (UID: "0d74ae5c-80fe-434b-9313-e42f60ae0518"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:08.984503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.984477    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-podinfo" pod "22e12023-6b7c-44a9-b1d1-016fd7e1c43b" (UID: "22e12023-6b7c-44a9-b1d1-016fd7e1c43b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:38:08.984503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.984489    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-podinfo" pod "0d74ae5c-80fe-434b-9313-e42f60ae0518" (UID: "0d74ae5c-80fe-434b-9313-e42f60ae0518"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:38:08.984503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.984498    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-kube-api-access-2s2bv" pod "22e12023-6b7c-44a9-b1d1-016fd7e1c43b" (UID: "22e12023-6b7c-44a9-b1d1-016fd7e1c43b"). InnerVolumeSpecName "kube-api-access-2s2bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:08.984682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.984522    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-kube-api-access-rx9pb" pod "0d74ae5c-80fe-434b-9313-e42f60ae0518" (UID: "0d74ae5c-80fe-434b-9313-e42f60ae0518"). InnerVolumeSpecName "kube-api-access-rx9pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:08.997419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.997395    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-envoy" pod "0d74ae5c-80fe-434b-9313-e42f60ae0518" (UID: "0d74ae5c-80fe-434b-9313-e42f60ae0518"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.997490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.997427    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-envoy" pod "22e12023-6b7c-44a9-b1d1-016fd7e1c43b" (UID: "22e12023-6b7c-44a9-b1d1-016fd7e1c43b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:38:08.997490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:08.997429    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-token" pod "22e12023-6b7c-44a9-b1d1-016fd7e1c43b" (UID: "22e12023-6b7c-44a9-b1d1-016fd7e1c43b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:09.058610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058578    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058608    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058621    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058632    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058640    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-kube-api-access-2s2bv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058649    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058658    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058666    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058673    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx9pb\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-kube-api-access-rx9pb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058682    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058697    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058705    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058713    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058721    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74ae5c-80fe-434b-9313-e42f60ae0518-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.058820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.058729    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e12023-6b7c-44a9-b1d1-016fd7e1c43b-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:09.548461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.548423    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"]
Nov 02 21:38:09.551255 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.551232    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6qkd9"]
Nov 02 21:38:09.660188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.660092    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"]
Nov 02 21:38:09.663676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.663654    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pd7dk"]
Nov 02 21:38:09.774895 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.774861    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 21:38:09.775306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.774919    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 21:38:09.821132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.820194    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gw8qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:38:09.821132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.820265    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:38:09.858984 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.858956    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"
Nov 02 21:38:09.859199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.858985    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb" event={"ID":"22e12023-6b7c-44a9-b1d1-016fd7e1c43b","Type":"ContainerDied","Data":"3b80bca1be3629fe372316a712b90a94dcd3ab98b32e7c3db0b70a6b053137f0"}
Nov 02 21:38:09.859199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.859037    4528 scope.go:122] "RemoveContainer" containerID="42e28435607b4942a4aec95ce65eccc6886d10510f3bb4f91a8628bd9dad4ac6"
Nov 02 21:38:09.862950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.862925    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r" event={"ID":"0d74ae5c-80fe-434b-9313-e42f60ae0518","Type":"ContainerDied","Data":"766173c76bd8f1d802f9b13fed96e37d8fc7b2b5b1f45ecdc104572e2a595ff1"}
Nov 02 21:38:09.863048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.862936    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"
Nov 02 21:38:09.869562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:09.869536    4528 scope.go:122] "RemoveContainer" containerID="ab0b54b75601731a7378fecdd44b2d116881189288c8623f67428a760de76e98"
Nov 02 21:38:10.034371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.034338    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"]
Nov 02 21:38:10.039862 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.039835    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gw8qb"]
Nov 02 21:38:10.150292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.150263    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"]
Nov 02 21:38:10.155301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.155270    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6js7r"]
Nov 02 21:38:10.775421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.775387    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 21:38:10.775828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.775440    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 21:38:10.932779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.932745    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" path="/var/lib/kubelet/pods/0d74ae5c-80fe-434b-9313-e42f60ae0518/volumes"
Nov 02 21:38:10.933278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.933264    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" path="/var/lib/kubelet/pods/22e12023-6b7c-44a9-b1d1-016fd7e1c43b/volumes"
Nov 02 21:38:10.933694 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.933680    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771361df-3dd7-4787-bf88-53a214521356" path="/var/lib/kubelet/pods/771361df-3dd7-4787-bf88-53a214521356/volumes"
Nov 02 21:38:10.934444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:10.934430    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" path="/var/lib/kubelet/pods/b32c3f43-49f5-412b-b226-f1d0ce064ac7/volumes"
Nov 02 21:38:11.774907 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:11.774873    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 02 21:38:11.775096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:11.774927    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 02 21:38:12.779631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:12.779598    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:38:12.780136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:12.779992    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:38:12.780660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:12.780639    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:38:14.118096 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:38:14.118054    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:38:16.033690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:16.033650    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"]
Nov 02 21:38:16.134164 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:16.134132    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"]
Nov 02 21:38:19.054724 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:19.054692    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"]
Nov 02 21:38:19.105433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:19.105395    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57442d13-jbzjn"]
Nov 02 21:38:20.913561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:20.913530    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c754c3-f2d5-4521-97a4-00b039d19756" path="/var/lib/kubelet/pods/c7c754c3-f2d5-4521-97a4-00b039d19756/volumes"
Nov 02 21:38:21.034812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.034771    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d" podUID="97cad693-11bb-4797-b96f-5a966171a90c" containerName="payload-processing" containerID="cri-o://b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a" gracePeriod=25
Nov 02 21:38:21.135195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.135155    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2" podUID="21ffd75a-0fa0-4960-8f10-4815301e82eb" containerName="payload-pre-processing" containerID="cri-o://5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9" gracePeriod=25
Nov 02 21:38:21.329291 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.329263    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:38:21.394984 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.394962    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:38:21.473013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.472905    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97cad693-11bb-4797-b96f-5a966171a90c-kube-api-access-qct74\" (UniqueName: \"kubernetes.io/projected/97cad693-11bb-4797-b96f-5a966171a90c-kube-api-access-qct74\") pod \"97cad693-11bb-4797-b96f-5a966171a90c\" (UID: \"97cad693-11bb-4797-b96f-5a966171a90c\") "
Nov 02 21:38:21.473013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.472984    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ffd75a-0fa0-4960-8f10-4815301e82eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ffd75a-0fa0-4960-8f10-4815301e82eb-plugins-config-volume\") pod \"21ffd75a-0fa0-4960-8f10-4815301e82eb\" (UID: \"21ffd75a-0fa0-4960-8f10-4815301e82eb\") "
Nov 02 21:38:21.473264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.473069    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ffd75a-0fa0-4960-8f10-4815301e82eb-kube-api-access-t8rfd\" (UniqueName: \"kubernetes.io/projected/21ffd75a-0fa0-4960-8f10-4815301e82eb-kube-api-access-t8rfd\") pod \"21ffd75a-0fa0-4960-8f10-4815301e82eb\" (UID: \"21ffd75a-0fa0-4960-8f10-4815301e82eb\") "
Nov 02 21:38:21.473330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.473275    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97cad693-11bb-4797-b96f-5a966171a90c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cad693-11bb-4797-b96f-5a966171a90c-plugins-config-volume\") pod \"97cad693-11bb-4797-b96f-5a966171a90c\" (UID: \"97cad693-11bb-4797-b96f-5a966171a90c\") "
Nov 02 21:38:21.474007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.473481    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ffd75a-0fa0-4960-8f10-4815301e82eb-plugins-config-volume" pod "21ffd75a-0fa0-4960-8f10-4815301e82eb" (UID: "21ffd75a-0fa0-4960-8f10-4815301e82eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:38:21.474007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.473818    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cad693-11bb-4797-b96f-5a966171a90c-plugins-config-volume" pod "97cad693-11bb-4797-b96f-5a966171a90c" (UID: "97cad693-11bb-4797-b96f-5a966171a90c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:38:21.474007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.473958    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ffd75a-0fa0-4960-8f10-4815301e82eb-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:21.474007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.473981    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cad693-11bb-4797-b96f-5a966171a90c-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:21.475429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.475403    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ffd75a-0fa0-4960-8f10-4815301e82eb-kube-api-access-t8rfd" pod "21ffd75a-0fa0-4960-8f10-4815301e82eb" (UID: "21ffd75a-0fa0-4960-8f10-4815301e82eb"). InnerVolumeSpecName "kube-api-access-t8rfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:21.475581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.475564    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cad693-11bb-4797-b96f-5a966171a90c-kube-api-access-qct74" pod "97cad693-11bb-4797-b96f-5a966171a90c" (UID: "97cad693-11bb-4797-b96f-5a966171a90c"). InnerVolumeSpecName "kube-api-access-qct74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:21.575730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.575693    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qct74\" (UniqueName: \"kubernetes.io/projected/97cad693-11bb-4797-b96f-5a966171a90c-kube-api-access-qct74\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:21.575730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.575729    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8rfd\" (UniqueName: \"kubernetes.io/projected/21ffd75a-0fa0-4960-8f10-4815301e82eb-kube-api-access-t8rfd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:21.960548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.960511    4528 generic.go:356] "Generic (PLEG): container finished" podID="97cad693-11bb-4797-b96f-5a966171a90c" containerID="b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a" exitCode=0
Nov 02 21:38:21.961063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.960555    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"
Nov 02 21:38:21.961063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.960597    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d" event={"ID":"97cad693-11bb-4797-b96f-5a966171a90c","Type":"ContainerDied","Data":"b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a"}
Nov 02 21:38:21.961063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.960643    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d" event={"ID":"97cad693-11bb-4797-b96f-5a966171a90c","Type":"ContainerDied","Data":"fde519b18c79ed33f232e244e5bdb717bed3334b4cce704a87f61f8705c3000e"}
Nov 02 21:38:21.961063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.960664    4528 scope.go:122] "RemoveContainer" containerID="b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a"
Nov 02 21:38:21.962201 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.962176    4528 generic.go:356] "Generic (PLEG): container finished" podID="21ffd75a-0fa0-4960-8f10-4815301e82eb" containerID="5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9" exitCode=0
Nov 02 21:38:21.962201 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.962210    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2" event={"ID":"21ffd75a-0fa0-4960-8f10-4815301e82eb","Type":"ContainerDied","Data":"5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9"}
Nov 02 21:38:21.962384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.962214    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"
Nov 02 21:38:21.962384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.962254    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2" event={"ID":"21ffd75a-0fa0-4960-8f10-4815301e82eb","Type":"ContainerDied","Data":"da6bb86feaeb54e43b5c5e758b158adf4609abb0802e487905f02c1e968db256"}
Nov 02 21:38:21.971377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.971358    4528 scope.go:122] "RemoveContainer" containerID="b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a"
Nov 02 21:38:21.971695 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:38:21.971678    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a\": container with ID starting with b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a not found: ID does not exist" containerID="b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a"
Nov 02 21:38:21.971757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.971702    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a"} err="failed to get container status \"b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a\": rpc error: code = NotFound desc = could not find container \"b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a\": container with ID starting with b90c667bfbde6a7fa8f7c44711fe209f54d90455205ee34c84662233407b634a not found: ID does not exist"
Nov 02 21:38:21.971757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.971716    4528 scope.go:122] "RemoveContainer" containerID="5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9"
Nov 02 21:38:21.979985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.979966    4528 scope.go:122] "RemoveContainer" containerID="5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9"
Nov 02 21:38:21.980287 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:38:21.980267    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9\": container with ID starting with 5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9 not found: ID does not exist" containerID="5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9"
Nov 02 21:38:21.980352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:21.980296    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9"} err="failed to get container status \"5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9\": rpc error: code = NotFound desc = could not find container \"5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9\": container with ID starting with 5c98832e09527dcb511c1689fcab403411056491f08cbb41b60a6059984dc2b9 not found: ID does not exist"
Nov 02 21:38:22.131172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:22.131142    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"]
Nov 02 21:38:22.133906 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:22.133883    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7b699ddbc9-bgzt2"]
Nov 02 21:38:22.242676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:22.242609    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"]
Nov 02 21:38:22.246628 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:22.246604    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-c9875d6d5-rcr7d"]
Nov 02 21:38:22.914239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:22.914206    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ffd75a-0fa0-4960-8f10-4815301e82eb" path="/var/lib/kubelet/pods/21ffd75a-0fa0-4960-8f10-4815301e82eb/volumes"
Nov 02 21:38:22.914640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:22.914626    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97cad693-11bb-4797-b96f-5a966171a90c" path="/var/lib/kubelet/pods/97cad693-11bb-4797-b96f-5a966171a90c/volumes"
Nov 02 21:38:25.206389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:25.206356    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:38:28.948451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.948419    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"]
Nov 02 21:38:28.951029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951005    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy"
Nov 02 21:38:28.951029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951028    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951045    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6271d098-7478-43e2-81ec-1f33de399d2d" containerName="test-curl-7471-3484f5"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951051    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6271d098-7478-43e2-81ec-1f33de399d2d" containerName="test-curl-7471-3484f5"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951076    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951083    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951094    4528 cpu_manager.go:414] "containerMap: removing container" podUID="21ffd75a-0fa0-4960-8f10-4815301e82eb" containerName="payload-pre-processing"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951099    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ffd75a-0fa0-4960-8f10-4815301e82eb" containerName="payload-pre-processing"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951128    4528 cpu_manager.go:414] "containerMap: removing container" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951135    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951146    4528 cpu_manager.go:414] "containerMap: removing container" podUID="aa666493-4f89-4a72-8a3e-ffc854458af3" containerName="payload-processing"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951150    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa666493-4f89-4a72-8a3e-ffc854458af3" containerName="payload-processing"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951156    4528 cpu_manager.go:414] "containerMap: removing container" podUID="97cad693-11bb-4797-b96f-5a966171a90c" containerName="payload-processing"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951160    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cad693-11bb-4797-b96f-5a966171a90c" containerName="payload-processing"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951167    4528 cpu_manager.go:414] "containerMap: removing container" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951174    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy"
Nov 02 21:38:28.951246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951254    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa666493-4f89-4a72-8a3e-ffc854458af3" containerName="payload-processing"
Nov 02 21:38:28.951683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951264    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cad693-11bb-4797-b96f-5a966171a90c" containerName="payload-processing"
Nov 02 21:38:28.951683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951270    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e12023-6b7c-44a9-b1d1-016fd7e1c43b" containerName="istio-proxy"
Nov 02 21:38:28.951683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951276    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d74ae5c-80fe-434b-9313-e42f60ae0518" containerName="istio-proxy"
Nov 02 21:38:28.951683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951282    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6271d098-7478-43e2-81ec-1f33de399d2d" containerName="test-curl-7471-3484f5"
Nov 02 21:38:28.951683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951291    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32c3f43-49f5-412b-b226-f1d0ce064ac7" containerName="istio-proxy"
Nov 02 21:38:28.951683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951297    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ffd75a-0fa0-4960-8f10-4815301e82eb" containerName="payload-pre-processing"
Nov 02 21:38:28.951683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.951303    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="771361df-3dd7-4787-bf88-53a214521356" containerName="istio-proxy"
Nov 02 21:38:28.954753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:28.954729    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.030431 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.030397    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"]
Nov 02 21:38:29.057402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.057375    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e12f99-openshift-default-dockercfg-6bc99\""
Nov 02 21:38:29.109208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.107291    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-certs\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.109208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.107396    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3734a586-cc32-4840-af2a-0d7d4b63bc46-istiod-ca-cert\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.109208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.107425    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-token\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.109208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.107450    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfqp\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-kube-api-access-4sfqp\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.109208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.107485    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-credential-socket\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.109208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.107591    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-envoy\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.109208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.107767    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-podinfo\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.109208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.107792    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-socket\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.109208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.107924    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-data\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.208762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.208673    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-envoy\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.208952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.208771    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-podinfo\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.208952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.208801    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-socket\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.208952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.208879    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-data\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.208952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.208920    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-certs\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.209274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.208992    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3734a586-cc32-4840-af2a-0d7d4b63bc46-istiod-ca-cert\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.209274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.209018    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-token\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.209274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.209043    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfqp\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-kube-api-access-4sfqp\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.209274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.209074    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-credential-socket\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.209274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.209207    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-socket\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.209575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.209285    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-data\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.209639 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.209573    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-credential-socket\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.209639 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.209617    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-certs\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.209810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.209785    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3734a586-cc32-4840-af2a-0d7d4b63bc46-istiod-ca-cert\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.211384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.211359    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-envoy\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.211384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.211377    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-podinfo\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.280414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.280377    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-token\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.282224 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.282196    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfqp\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-kube-api-access-4sfqp\") pod \"e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") " pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:29.573360 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:29.573323    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:30.005869 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:38:30.005842    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3734a586_cc32_4840_af2a_0d7d4b63bc46.slice/crio-0da1068b3ddc5fa7b43bf6b7216be777d6ba8ac2a9eb9e0588504fb741ea1cc0 WatchSource:0}: Error finding container 0da1068b3ddc5fa7b43bf6b7216be777d6ba8ac2a9eb9e0588504fb741ea1cc0: Status 404 returned error can't find the container with id 0da1068b3ddc5fa7b43bf6b7216be777d6ba8ac2a9eb9e0588504fb741ea1cc0
Nov 02 21:38:30.007958 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:30.007937    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"]
Nov 02 21:38:30.008476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:30.008446    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:38:30.008547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:30.008504    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:38:30.008547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:30.008531    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:38:30.046870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:30.046842    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" event={"ID":"3734a586-cc32-4840-af2a-0d7d4b63bc46","Type":"ContainerStarted","Data":"0da1068b3ddc5fa7b43bf6b7216be777d6ba8ac2a9eb9e0588504fb741ea1cc0"}
Nov 02 21:38:31.054645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:31.054610    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" event={"ID":"3734a586-cc32-4840-af2a-0d7d4b63bc46","Type":"ContainerStarted","Data":"4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c"}
Nov 02 21:38:31.272156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:31.272048    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" podStartSLOduration=3.272032664 podStartE2EDuration="3.272032664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:31.221655372 +0000 UTC m=+1798.485191690" watchObservedRunningTime="2026-11-02 21:38:31.272032664 +0000 UTC m=+1798.535568982"
Nov 02 21:38:31.574284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:31.574255    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:31.579843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:31.579819    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:32.061269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:32.061240    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:32.062228 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:32.062181    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:38:34.097685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:34.097648    4528 scope.go:122] "RemoveContainer" containerID="90700e2b8079bdc3c1cf5e48ded922f0c2e3edbe1ec482a6e9dcdfcaf0b5cf05"
Nov 02 21:38:37.534723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.534690    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"]
Nov 02 21:38:37.540227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.540208    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:37.592003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.591972    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"]
Nov 02 21:38:37.613987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.613948    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9tw\" (UniqueName: \"kubernetes.io/projected/4951470f-4373-4bce-a1db-f95395dbc27e-kube-api-access-8l9tw\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g\" (UID: \"4951470f-4373-4bce-a1db-f95395dbc27e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:37.615040 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.615013    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4951470f-4373-4bce-a1db-f95395dbc27e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g\" (UID: \"4951470f-4373-4bce-a1db-f95395dbc27e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:37.691264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.676356    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e12f99\""
Nov 02 21:38:37.691264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.676520    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e12f99-dockercfg-mwlw5\""
Nov 02 21:38:37.719384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.719091    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9tw\" (UniqueName: \"kubernetes.io/projected/4951470f-4373-4bce-a1db-f95395dbc27e-kube-api-access-8l9tw\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g\" (UID: \"4951470f-4373-4bce-a1db-f95395dbc27e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:37.719384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.719199    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4951470f-4373-4bce-a1db-f95395dbc27e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g\" (UID: \"4951470f-4373-4bce-a1db-f95395dbc27e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:37.721512 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.721483    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4951470f-4373-4bce-a1db-f95395dbc27e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g\" (UID: \"4951470f-4373-4bce-a1db-f95395dbc27e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:37.721665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.721605    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"]
Nov 02 21:38:37.726702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.726678    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:37.764146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.764088    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"]
Nov 02 21:38:37.778118 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.778080    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9tw\" (UniqueName: \"kubernetes.io/projected/4951470f-4373-4bce-a1db-f95395dbc27e-kube-api-access-8l9tw\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g\" (UID: \"4951470f-4373-4bce-a1db-f95395dbc27e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:37.820186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.820093    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b46a41-5785-4ed1-ba52-eae364782587-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt\" (UID: \"91b46a41-5785-4ed1-ba52-eae364782587\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:37.820313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.820213    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfn8f\" (UniqueName: \"kubernetes.io/projected/91b46a41-5785-4ed1-ba52-eae364782587-kube-api-access-gfn8f\") pod \"payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt\" (UID: \"91b46a41-5785-4ed1-ba52-eae364782587\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:37.853409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.853380    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:37.922144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.922073    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfn8f\" (UniqueName: \"kubernetes.io/projected/91b46a41-5785-4ed1-ba52-eae364782587-kube-api-access-gfn8f\") pod \"payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt\" (UID: \"91b46a41-5785-4ed1-ba52-eae364782587\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:37.926475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.926444    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b46a41-5785-4ed1-ba52-eae364782587-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt\" (UID: \"91b46a41-5785-4ed1-ba52-eae364782587\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:37.927322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.927302    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b46a41-5785-4ed1-ba52-eae364782587-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt\" (UID: \"91b46a41-5785-4ed1-ba52-eae364782587\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:37.979786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:37.979754    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfn8f\" (UniqueName: \"kubernetes.io/projected/91b46a41-5785-4ed1-ba52-eae364782587-kube-api-access-gfn8f\") pod \"payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt\" (UID: \"91b46a41-5785-4ed1-ba52-eae364782587\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:38.042919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:38.042893    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:38.224704 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:38:38.224674    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4951470f_4373_4bce_a1db_f95395dbc27e.slice/crio-32d0811746aa056513df35e8bc08350cc7dcc004e8081a0080e8319192373834 WatchSource:0}: Error finding container 32d0811746aa056513df35e8bc08350cc7dcc004e8081a0080e8319192373834: Status 404 returned error can't find the container with id 32d0811746aa056513df35e8bc08350cc7dcc004e8081a0080e8319192373834
Nov 02 21:38:38.228056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:38.228037    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"]
Nov 02 21:38:38.410323 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:38:38.410294    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b46a41_5785_4ed1_ba52_eae364782587.slice/crio-2409dccc6096d373526d1e573d4f547baed836a09b976097b8949f47d9d61a31 WatchSource:0}: Error finding container 2409dccc6096d373526d1e573d4f547baed836a09b976097b8949f47d9d61a31: Status 404 returned error can't find the container with id 2409dccc6096d373526d1e573d4f547baed836a09b976097b8949f47d9d61a31
Nov 02 21:38:38.412894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:38.412872    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"]
Nov 02 21:38:39.111532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:39.111496    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt" event={"ID":"91b46a41-5785-4ed1-ba52-eae364782587","Type":"ContainerStarted","Data":"cf936d0bebbda525f117f16a7ff30b74a174ce459879a9e137717985382d1bc0"}
Nov 02 21:38:39.112001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:39.111547    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt" event={"ID":"91b46a41-5785-4ed1-ba52-eae364782587","Type":"ContainerStarted","Data":"2409dccc6096d373526d1e573d4f547baed836a09b976097b8949f47d9d61a31"}
Nov 02 21:38:39.112001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:39.111649    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:39.113211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:39.113189    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g" event={"ID":"4951470f-4373-4bce-a1db-f95395dbc27e","Type":"ContainerStarted","Data":"c60f212b01922256651e27b46672b0383d932d723e8b10af02ecfb424889f6fa"}
Nov 02 21:38:39.113350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:39.113224    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g" event={"ID":"4951470f-4373-4bce-a1db-f95395dbc27e","Type":"ContainerStarted","Data":"32d0811746aa056513df35e8bc08350cc7dcc004e8081a0080e8319192373834"}
Nov 02 21:38:39.113350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:39.113281    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:39.227388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:39.227338    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt" podStartSLOduration=2.22732442 podStartE2EDuration="2.22732442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:39.224688869 +0000 UTC m=+1806.488225187" watchObservedRunningTime="2026-11-02 21:38:39.22732442 +0000 UTC m=+1806.490860786"
Nov 02 21:38:39.351787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:39.349266    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g" podStartSLOduration=2.349247015 podStartE2EDuration="2.349247015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:39.346474225 +0000 UTC m=+1806.610010610" watchObservedRunningTime="2026-11-02 21:38:39.349247015 +0000 UTC m=+1806.612783334"
Nov 02 21:38:42.629056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:42.629020    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"]
Nov 02 21:38:42.739951 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:42.738473    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"]
Nov 02 21:38:44.118819 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:38:44.118781    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:38:47.629478 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:47.629424    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65" podUID="23866e02-cd31-4613-bd0d-8f7eb7ed6550" containerName="payload-processing" containerID="cri-o://7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362" gracePeriod=25
Nov 02 21:38:47.739389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:47.739353    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l" podUID="a78ce437-a092-4025-9da9-9069567c86d9" containerName="payload-pre-processing" containerID="cri-o://995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2" gracePeriod=25
Nov 02 21:38:47.908102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:47.908021    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:38:47.959274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:47.959245    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23866e02-cd31-4613-bd0d-8f7eb7ed6550-kube-api-access-vbgc7\" (UniqueName: \"kubernetes.io/projected/23866e02-cd31-4613-bd0d-8f7eb7ed6550-kube-api-access-vbgc7\") pod \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\" (UID: \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\") "
Nov 02 21:38:47.959433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:47.959357    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23866e02-cd31-4613-bd0d-8f7eb7ed6550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23866e02-cd31-4613-bd0d-8f7eb7ed6550-plugins-config-volume\") pod \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\" (UID: \"23866e02-cd31-4613-bd0d-8f7eb7ed6550\") "
Nov 02 21:38:47.959787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:47.959758    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23866e02-cd31-4613-bd0d-8f7eb7ed6550-plugins-config-volume" pod "23866e02-cd31-4613-bd0d-8f7eb7ed6550" (UID: "23866e02-cd31-4613-bd0d-8f7eb7ed6550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:38:47.959924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:47.959881    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23866e02-cd31-4613-bd0d-8f7eb7ed6550-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:47.962200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:47.962162    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23866e02-cd31-4613-bd0d-8f7eb7ed6550-kube-api-access-vbgc7" pod "23866e02-cd31-4613-bd0d-8f7eb7ed6550" (UID: "23866e02-cd31-4613-bd0d-8f7eb7ed6550"). InnerVolumeSpecName "kube-api-access-vbgc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:48.001304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.001283    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:38:48.130136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.102813    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78ce437-a092-4025-9da9-9069567c86d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78ce437-a092-4025-9da9-9069567c86d9-plugins-config-volume\") pod \"a78ce437-a092-4025-9da9-9069567c86d9\" (UID: \"a78ce437-a092-4025-9da9-9069567c86d9\") "
Nov 02 21:38:48.130136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.102970    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78ce437-a092-4025-9da9-9069567c86d9-kube-api-access-8skdn\" (UniqueName: \"kubernetes.io/projected/a78ce437-a092-4025-9da9-9069567c86d9-kube-api-access-8skdn\") pod \"a78ce437-a092-4025-9da9-9069567c86d9\" (UID: \"a78ce437-a092-4025-9da9-9069567c86d9\") "
Nov 02 21:38:48.157135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.131278    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbgc7\" (UniqueName: \"kubernetes.io/projected/23866e02-cd31-4613-bd0d-8f7eb7ed6550-kube-api-access-vbgc7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:48.164287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.157555    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78ce437-a092-4025-9da9-9069567c86d9-plugins-config-volume" pod "a78ce437-a092-4025-9da9-9069567c86d9" (UID: "a78ce437-a092-4025-9da9-9069567c86d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:38:48.176029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.175983    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78ce437-a092-4025-9da9-9069567c86d9-kube-api-access-8skdn" pod "a78ce437-a092-4025-9da9-9069567c86d9" (UID: "a78ce437-a092-4025-9da9-9069567c86d9"). InnerVolumeSpecName "kube-api-access-8skdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:38:48.209286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.209248    4528 generic.go:356] "Generic (PLEG): container finished" podID="23866e02-cd31-4613-bd0d-8f7eb7ed6550" containerID="7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362" exitCode=0
Nov 02 21:38:48.209592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.209305    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"
Nov 02 21:38:48.209703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.209326    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65" event={"ID":"23866e02-cd31-4613-bd0d-8f7eb7ed6550","Type":"ContainerDied","Data":"7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362"}
Nov 02 21:38:48.209774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.209719    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65" event={"ID":"23866e02-cd31-4613-bd0d-8f7eb7ed6550","Type":"ContainerDied","Data":"effa4fd556ded532607c3782970fe95a42f675df6c45614aa0785225ef34bcbf"}
Nov 02 21:38:48.210294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.210268    4528 scope.go:122] "RemoveContainer" containerID="7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362"
Nov 02 21:38:48.213494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.213472    4528 generic.go:356] "Generic (PLEG): container finished" podID="a78ce437-a092-4025-9da9-9069567c86d9" containerID="995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2" exitCode=0
Nov 02 21:38:48.213689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.213659    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l" event={"ID":"a78ce437-a092-4025-9da9-9069567c86d9","Type":"ContainerDied","Data":"995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2"}
Nov 02 21:38:48.213857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.213829    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l" event={"ID":"a78ce437-a092-4025-9da9-9069567c86d9","Type":"ContainerDied","Data":"93cfdc4ff538373a7dd6357a3c0de1e20479b3fa7fcec315ec40e209a1fa4eac"}
Nov 02 21:38:48.214066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.214048    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"
Nov 02 21:38:48.229199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.229176    4528 scope.go:122] "RemoveContainer" containerID="7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362"
Nov 02 21:38:48.229856 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:38:48.229832    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362\": container with ID starting with 7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362 not found: ID does not exist" containerID="7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362"
Nov 02 21:38:48.229957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.229866    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362"} err="failed to get container status \"7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362\": rpc error: code = NotFound desc = could not find container \"7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362\": container with ID starting with 7aa7d2302e46a0aecb357e9d3530abb2e2b5820aaeb264f51c6c4bcbb47ed362 not found: ID does not exist"
Nov 02 21:38:48.229957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.229888    4528 scope.go:122] "RemoveContainer" containerID="995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2"
Nov 02 21:38:48.232681 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.232659    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8skdn\" (UniqueName: \"kubernetes.io/projected/a78ce437-a092-4025-9da9-9069567c86d9-kube-api-access-8skdn\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:48.232760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.232682    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78ce437-a092-4025-9da9-9069567c86d9-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:38:48.239192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.239170    4528 scope.go:122] "RemoveContainer" containerID="995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2"
Nov 02 21:38:48.239508 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:38:48.239487    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2\": container with ID starting with 995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2 not found: ID does not exist" containerID="995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2"
Nov 02 21:38:48.239575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.239516    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2"} err="failed to get container status \"995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2\": rpc error: code = NotFound desc = could not find container \"995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2\": container with ID starting with 995c4f46f73210165016a38c75d0dd26a8bea7852edd114bfd7de70162e8a6c2 not found: ID does not exist"
Nov 02 21:38:48.374730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.374693    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"]
Nov 02 21:38:48.380003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.379976    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-6898c5787-8ww65"]
Nov 02 21:38:48.489321 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.489241    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"]
Nov 02 21:38:48.492615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.492591    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-774c866c-zbc6l"]
Nov 02 21:38:48.927746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.927663    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23866e02-cd31-4613-bd0d-8f7eb7ed6550" path="/var/lib/kubelet/pods/23866e02-cd31-4613-bd0d-8f7eb7ed6550/volumes"
Nov 02 21:38:48.928282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:48.928264    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78ce437-a092-4025-9da9-9069567c86d9" path="/var/lib/kubelet/pods/a78ce437-a092-4025-9da9-9069567c86d9/volumes"
Nov 02 21:38:50.127134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:50.127085    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:38:50.127721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:50.127701    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:38:56.772247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:56.772212    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"]
Nov 02 21:38:56.772675 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:56.772434    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy" containerID="cri-o://2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d" gracePeriod=30
Nov 02 21:38:58.307062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:58.307024    4528 generic.go:356] "Generic (PLEG): container finished" podID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerID="75d747d1bf64e973afbb06870b021192b3ea0ae365b6b9f41fa18ef040a1a4b0" exitCode=0
Nov 02 21:38:58.307501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:58.307097    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" event={"ID":"7a206f69-8091-412f-9a80-e89cd60e50a5","Type":"ContainerDied","Data":"75d747d1bf64e973afbb06870b021192b3ea0ae365b6b9f41fa18ef040a1a4b0"}
Nov 02 21:38:58.307501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:58.307490    4528 scope.go:122] "RemoveContainer" containerID="75d747d1bf64e973afbb06870b021192b3ea0ae365b6b9f41fa18ef040a1a4b0"
Nov 02 21:38:58.308840 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:58.308811    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:38:58.308979 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:58.308890    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:38:58.309051 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:58.308958    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:38:59.335687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:59.335654    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" event={"ID":"7a206f69-8091-412f-9a80-e89cd60e50a5","Type":"ContainerStarted","Data":"102c0939bd138096ad2b395c599010edb1c5aacab26a9d7f59a7ccef12e2d613"}
Nov 02 21:38:59.645339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:59.645246    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:38:59.646662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:59.646634    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 02 21:38:59.646790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:38:59.646680    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 02 21:39:00.645120 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:00.645072    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 02 21:39:00.645510 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:00.645145    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 02 21:39:01.645065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:01.645028    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 02 21:39:01.645270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:01.645082    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 02 21:39:01.964166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:01.964053    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 21:39:01.964166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:01.964105    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 21:39:02.089146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.089086    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:39:02.131508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.131479    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:39:02.152329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152302    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-socket\") pod \"4531e47f-f6df-48cd-948d-78b67df65b79\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") "
Nov 02 21:39:02.152477 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152357    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-data\") pod \"4531e47f-f6df-48cd-948d-78b67df65b79\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") "
Nov 02 21:39:02.152477 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152395    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-istio-token\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-istio-token\") pod \"4531e47f-f6df-48cd-948d-78b67df65b79\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") "
Nov 02 21:39:02.152477 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152428    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-envoy\") pod \"4531e47f-f6df-48cd-948d-78b67df65b79\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") "
Nov 02 21:39:02.152581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152536    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-credential-socket\") pod \"4531e47f-f6df-48cd-948d-78b67df65b79\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") "
Nov 02 21:39:02.152581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152573    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4531e47f-f6df-48cd-948d-78b67df65b79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531e47f-f6df-48cd-948d-78b67df65b79-istio-podinfo\") pod \"4531e47f-f6df-48cd-948d-78b67df65b79\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") "
Nov 02 21:39:02.152643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152583    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-socket" pod "4531e47f-f6df-48cd-948d-78b67df65b79" (UID: "4531e47f-f6df-48cd-948d-78b67df65b79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:39:02.152643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152610    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-certs\") pod \"4531e47f-f6df-48cd-948d-78b67df65b79\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") "
Nov 02 21:39:02.152707 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152662    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-kube-api-access-rzglz\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-kube-api-access-rzglz\") pod \"4531e47f-f6df-48cd-948d-78b67df65b79\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") "
Nov 02 21:39:02.152741 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.152708    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4531e47f-f6df-48cd-948d-78b67df65b79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531e47f-f6df-48cd-948d-78b67df65b79-istiod-ca-cert\") pod \"4531e47f-f6df-48cd-948d-78b67df65b79\" (UID: \"4531e47f-f6df-48cd-948d-78b67df65b79\") "
Nov 02 21:39:02.153131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.153066    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-data" pod "4531e47f-f6df-48cd-948d-78b67df65b79" (UID: "4531e47f-f6df-48cd-948d-78b67df65b79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:39:02.153249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.153147    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:02.153249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.153156    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-certs" pod "4531e47f-f6df-48cd-948d-78b67df65b79" (UID: "4531e47f-f6df-48cd-948d-78b67df65b79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:39:02.153249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.153167    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-credential-socket" pod "4531e47f-f6df-48cd-948d-78b67df65b79" (UID: "4531e47f-f6df-48cd-948d-78b67df65b79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:39:02.153430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.153407    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4531e47f-f6df-48cd-948d-78b67df65b79-istiod-ca-cert" pod "4531e47f-f6df-48cd-948d-78b67df65b79" (UID: "4531e47f-f6df-48cd-948d-78b67df65b79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:39:02.154820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.154800    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-envoy" pod "4531e47f-f6df-48cd-948d-78b67df65b79" (UID: "4531e47f-f6df-48cd-948d-78b67df65b79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:39:02.155088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.155066    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-istio-token" pod "4531e47f-f6df-48cd-948d-78b67df65b79" (UID: "4531e47f-f6df-48cd-948d-78b67df65b79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:39:02.155175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.155151    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4531e47f-f6df-48cd-948d-78b67df65b79-istio-podinfo" pod "4531e47f-f6df-48cd-948d-78b67df65b79" (UID: "4531e47f-f6df-48cd-948d-78b67df65b79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:39:02.155315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.155296    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-kube-api-access-rzglz" pod "4531e47f-f6df-48cd-948d-78b67df65b79" (UID: "4531e47f-f6df-48cd-948d-78b67df65b79"). InnerVolumeSpecName "kube-api-access-rzglz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:39:02.254576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.254492    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:02.254805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.254787    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531e47f-f6df-48cd-948d-78b67df65b79-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:02.254945 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.254931    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:02.255063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.255049    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzglz\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-kube-api-access-rzglz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:02.255226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.255210    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531e47f-f6df-48cd-948d-78b67df65b79-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:02.255361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.255347    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:02.255470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.255457    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4531e47f-f6df-48cd-948d-78b67df65b79-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:02.255608 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.255572    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531e47f-f6df-48cd-948d-78b67df65b79-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:02.370499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.370464    4528 generic.go:356] "Generic (PLEG): container finished" podID="4531e47f-f6df-48cd-948d-78b67df65b79" containerID="2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d" exitCode=0
Nov 02 21:39:02.370699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.370543    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" event={"ID":"4531e47f-f6df-48cd-948d-78b67df65b79","Type":"ContainerDied","Data":"2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d"}
Nov 02 21:39:02.370699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.370606    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg" event={"ID":"4531e47f-f6df-48cd-948d-78b67df65b79","Type":"ContainerDied","Data":"8049714bf81f83e822532fd88f6bf3ab88637adaaf01abe809279bf28f66d777"}
Nov 02 21:39:02.370699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.370632    4528 scope.go:122] "RemoveContainer" containerID="2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d"
Nov 02 21:39:02.371197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.371172    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"
Nov 02 21:39:02.382291 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.382254    4528 scope.go:122] "RemoveContainer" containerID="2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d"
Nov 02 21:39:02.382566 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:39:02.382543    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d\": container with ID starting with 2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d not found: ID does not exist" containerID="2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d"
Nov 02 21:39:02.382647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.382578    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d"} err="failed to get container status \"2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d\": rpc error: code = NotFound desc = could not find container \"2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d\": container with ID starting with 2044e0b72f29248591c617b59567622da87a7154fc53350cb20bbd459cb9262d not found: ID does not exist"
Nov 02 21:39:02.399389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.399369    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:39:02.538680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.538644    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"]
Nov 02 21:39:02.544866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.544839    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-79bdg"]
Nov 02 21:39:02.644783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.644741    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 02 21:39:02.644990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.644806    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 02 21:39:02.934988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:02.934916    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" path="/var/lib/kubelet/pods/4531e47f-f6df-48cd-948d-78b67df65b79/volumes"
Nov 02 21:39:03.644638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:03.644605    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 02 21:39:03.644813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:03.644659    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 02 21:39:04.649167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:04.649137    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:39:04.650157 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:04.650139    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:39:06.391659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.391630    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"]
Nov 02 21:39:06.392260 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.392243    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a78ce437-a092-4025-9da9-9069567c86d9" containerName="payload-pre-processing"
Nov 02 21:39:06.392348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.392261    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78ce437-a092-4025-9da9-9069567c86d9" containerName="payload-pre-processing"
Nov 02 21:39:06.392348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.392277    4528 cpu_manager.go:414] "containerMap: removing container" podUID="23866e02-cd31-4613-bd0d-8f7eb7ed6550" containerName="payload-processing"
Nov 02 21:39:06.392348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.392282    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23866e02-cd31-4613-bd0d-8f7eb7ed6550" containerName="payload-processing"
Nov 02 21:39:06.392348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.392309    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy"
Nov 02 21:39:06.392348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.392313    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy"
Nov 02 21:39:06.392521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.392395    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4531e47f-f6df-48cd-948d-78b67df65b79" containerName="istio-proxy"
Nov 02 21:39:06.392521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.392407    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78ce437-a092-4025-9da9-9069567c86d9" containerName="payload-pre-processing"
Nov 02 21:39:06.392521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.392415    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="23866e02-cd31-4613-bd0d-8f7eb7ed6550" containerName="payload-processing"
Nov 02 21:39:06.396978 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.396943    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.472101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.472052    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"]
Nov 02 21:39:06.514593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.514560    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ztxl\" (UniqueName: \"kubernetes.io/projected/841b76a5-26ea-40f5-9e45-51d7609e670b-kube-api-access-8ztxl\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-r78zv\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.514737 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.514616    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-r78zv\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.514791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.514746    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-r78zv\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.517422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.517399    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:39:06.517550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.517423    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:39:06.517550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.517445    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:39:06.517550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.517483    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:39:06.638811 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.638765    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ztxl\" (UniqueName: \"kubernetes.io/projected/841b76a5-26ea-40f5-9e45-51d7609e670b-kube-api-access-8ztxl\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-r78zv\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.639000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.638860    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-r78zv\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.639000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.638952    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-r78zv\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.640208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.640181    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-r78zv\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.686162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.686040    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-r78zv\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.697543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.697508    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ztxl\" (UniqueName: \"kubernetes.io/projected/841b76a5-26ea-40f5-9e45-51d7609e670b-kube-api-access-8ztxl\") pod \"maas-api-revoke-keys-e2e-mt-70efbbab-r78zv\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:06.710197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:06.710155    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:07.082713 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:39:07.082683    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841b76a5_26ea_40f5_9e45_51d7609e670b.slice/crio-f3fd82136e8f861e07b42fce4b4ffa3471bab73526d5a35574a4c61e19a98f8c WatchSource:0}: Error finding container f3fd82136e8f861e07b42fce4b4ffa3471bab73526d5a35574a4c61e19a98f8c: Status 404 returned error can't find the container with id f3fd82136e8f861e07b42fce4b4ffa3471bab73526d5a35574a4c61e19a98f8c
Nov 02 21:39:07.088260 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.088102    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"]
Nov 02 21:39:07.402330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.402258    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv" event={"ID":"841b76a5-26ea-40f5-9e45-51d7609e670b","Type":"ContainerStarted","Data":"f3fd82136e8f861e07b42fce4b4ffa3471bab73526d5a35574a4c61e19a98f8c"}
Nov 02 21:39:07.410093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.410067    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"]
Nov 02 21:39:07.415179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.415163    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:07.467847 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.467820    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-54e844\""
Nov 02 21:39:07.468015 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.467951    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-54e844-dockercfg-qmltf\""
Nov 02 21:39:07.468189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.468170    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"]
Nov 02 21:39:07.547943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.547901    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzt8x\" (UniqueName: \"kubernetes.io/projected/84e8162f-c535-478b-832c-53d433bf98af-kube-api-access-gzt8x\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq\" (UID: \"84e8162f-c535-478b-832c-53d433bf98af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:07.548082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.548053    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e8162f-c535-478b-832c-53d433bf98af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq\" (UID: \"84e8162f-c535-478b-832c-53d433bf98af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:07.649257 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.649229    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e8162f-c535-478b-832c-53d433bf98af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq\" (UID: \"84e8162f-c535-478b-832c-53d433bf98af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:07.649393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.649343    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzt8x\" (UniqueName: \"kubernetes.io/projected/84e8162f-c535-478b-832c-53d433bf98af-kube-api-access-gzt8x\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq\" (UID: \"84e8162f-c535-478b-832c-53d433bf98af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:07.649900 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.649874    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e8162f-c535-478b-832c-53d433bf98af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq\" (UID: \"84e8162f-c535-478b-832c-53d433bf98af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:07.706492 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.706426    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzt8x\" (UniqueName: \"kubernetes.io/projected/84e8162f-c535-478b-832c-53d433bf98af-kube-api-access-gzt8x\") pod \"payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq\" (UID: \"84e8162f-c535-478b-832c-53d433bf98af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:07.735295 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:07.735265    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:08.105865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.105836    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"]
Nov 02 21:39:08.150886 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:39:08.150857    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e8162f_c535_478b_832c_53d433bf98af.slice/crio-173ba9387b7bb18f707a95295784e8add875e458cc70a13f689a240eef1249ac WatchSource:0}: Error finding container 173ba9387b7bb18f707a95295784e8add875e458cc70a13f689a240eef1249ac: Status 404 returned error can't find the container with id 173ba9387b7bb18f707a95295784e8add875e458cc70a13f689a240eef1249ac
Nov 02 21:39:08.185879 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.185856    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"]
Nov 02 21:39:08.189876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.189861    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:08.243653 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.243621    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"]
Nov 02 21:39:08.256777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.256747    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqfcc\" (UniqueName: \"kubernetes.io/projected/29fd19f0-2a8c-4c26-8536-d02013df7a57-kube-api-access-vqfcc\") pod \"payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn\" (UID: \"29fd19f0-2a8c-4c26-8536-d02013df7a57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:08.256910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.256894    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fd19f0-2a8c-4c26-8536-d02013df7a57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn\" (UID: \"29fd19f0-2a8c-4c26-8536-d02013df7a57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:08.358098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.358009    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqfcc\" (UniqueName: \"kubernetes.io/projected/29fd19f0-2a8c-4c26-8536-d02013df7a57-kube-api-access-vqfcc\") pod \"payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn\" (UID: \"29fd19f0-2a8c-4c26-8536-d02013df7a57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:08.358098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.358081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fd19f0-2a8c-4c26-8536-d02013df7a57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn\" (UID: \"29fd19f0-2a8c-4c26-8536-d02013df7a57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:08.358782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.358760    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fd19f0-2a8c-4c26-8536-d02013df7a57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn\" (UID: \"29fd19f0-2a8c-4c26-8536-d02013df7a57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:08.409402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.409368    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq" event={"ID":"84e8162f-c535-478b-832c-53d433bf98af","Type":"ContainerStarted","Data":"173ba9387b7bb18f707a95295784e8add875e458cc70a13f689a240eef1249ac"}
Nov 02 21:39:08.411054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.411031    4528 generic.go:356] "Generic (PLEG): container finished" podID="841b76a5-26ea-40f5-9e45-51d7609e670b" containerID="ac685e660fb0d21325b3fe53178babd3f737698c90121c8b19e78d5df99e0c89" exitCode=0
Nov 02 21:39:08.411153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.411074    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv" event={"ID":"841b76a5-26ea-40f5-9e45-51d7609e670b","Type":"ContainerDied","Data":"ac685e660fb0d21325b3fe53178babd3f737698c90121c8b19e78d5df99e0c89"}
Nov 02 21:39:08.415818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.415796    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqfcc\" (UniqueName: \"kubernetes.io/projected/29fd19f0-2a8c-4c26-8536-d02013df7a57-kube-api-access-vqfcc\") pod \"payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn\" (UID: \"29fd19f0-2a8c-4c26-8536-d02013df7a57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:08.503087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.503057    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:08.869935 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:39:08.869901    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fd19f0_2a8c_4c26_8536_d02013df7a57.slice/crio-f83edb90d689f8103b5a2bc313b94eab2e035dd8adf4d0c9c8d405374e967ca2 WatchSource:0}: Error finding container f83edb90d689f8103b5a2bc313b94eab2e035dd8adf4d0c9c8d405374e967ca2: Status 404 returned error can't find the container with id f83edb90d689f8103b5a2bc313b94eab2e035dd8adf4d0c9c8d405374e967ca2
Nov 02 21:39:08.872745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:08.872719    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"]
Nov 02 21:39:09.453643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.453607    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn" event={"ID":"29fd19f0-2a8c-4c26-8536-d02013df7a57","Type":"ContainerStarted","Data":"f83edb90d689f8103b5a2bc313b94eab2e035dd8adf4d0c9c8d405374e967ca2"}
Nov 02 21:39:09.641313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.641285    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:09.775262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.775222    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841b76a5-26ea-40f5-9e45-51d7609e670b-kube-api-access-8ztxl\" (UniqueName: \"kubernetes.io/projected/841b76a5-26ea-40f5-9e45-51d7609e670b-kube-api-access-8ztxl\") pod \"841b76a5-26ea-40f5-9e45-51d7609e670b\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") "
Nov 02 21:39:09.775461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.775376    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-openshift-ca\") pod \"841b76a5-26ea-40f5-9e45-51d7609e670b\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") "
Nov 02 21:39:09.775537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.775463    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-xks-ca\") pod \"841b76a5-26ea-40f5-9e45-51d7609e670b\" (UID: \"841b76a5-26ea-40f5-9e45-51d7609e670b\") "
Nov 02 21:39:09.775837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.775805    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-openshift-ca" pod "841b76a5-26ea-40f5-9e45-51d7609e670b" (UID: "841b76a5-26ea-40f5-9e45-51d7609e670b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:39:09.776012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.775991    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:09.781512 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.781485    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-xks-ca" pod "841b76a5-26ea-40f5-9e45-51d7609e670b" (UID: "841b76a5-26ea-40f5-9e45-51d7609e670b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:39:09.783272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.783235    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841b76a5-26ea-40f5-9e45-51d7609e670b-kube-api-access-8ztxl" pod "841b76a5-26ea-40f5-9e45-51d7609e670b" (UID: "841b76a5-26ea-40f5-9e45-51d7609e670b"). InnerVolumeSpecName "kube-api-access-8ztxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:39:09.877713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.877681    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ztxl\" (UniqueName: \"kubernetes.io/projected/841b76a5-26ea-40f5-9e45-51d7609e670b-kube-api-access-8ztxl\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:09.877713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:09.877715    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841b76a5-26ea-40f5-9e45-51d7609e670b-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:10.461476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:10.461434    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq" event={"ID":"84e8162f-c535-478b-832c-53d433bf98af","Type":"ContainerStarted","Data":"65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb"}
Nov 02 21:39:10.461904 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:10.461523    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:10.463067 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:10.463040    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn" event={"ID":"29fd19f0-2a8c-4c26-8536-d02013df7a57","Type":"ContainerStarted","Data":"2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52"}
Nov 02 21:39:10.463188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:10.463139    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:10.464282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:10.464261    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv" event={"ID":"841b76a5-26ea-40f5-9e45-51d7609e670b","Type":"ContainerDied","Data":"f3fd82136e8f861e07b42fce4b4ffa3471bab73526d5a35574a4c61e19a98f8c"}
Nov 02 21:39:10.464373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:10.464283    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"
Nov 02 21:39:10.464373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:10.464294    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3fd82136e8f861e07b42fce4b4ffa3471bab73526d5a35574a4c61e19a98f8c"
Nov 02 21:39:10.577325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:10.577275    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq" podStartSLOduration=1.597082567 podStartE2EDuration="3.577261047s" totalImagesPullingTime="1.98017848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:10.575856009 +0000 UTC m=+1837.839392321" watchObservedRunningTime="2026-11-02 21:39:10.577261047 +0000 UTC m=+1837.840797364"
Nov 02 21:39:10.725293 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:10.722080    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn" podStartSLOduration=1.231770121 podStartE2EDuration="2.722058033s" totalImagesPullingTime="1.490287912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:10.700519027 +0000 UTC m=+1837.964055359" watchObservedRunningTime="2026-11-02 21:39:10.722058033 +0000 UTC m=+1837.985594356"
Nov 02 21:39:14.113179 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:39:14.113144    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:39:15.943824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:15.943793    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"]
Nov 02 21:39:16.049803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:16.049770    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"]
Nov 02 21:39:20.944128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:20.944068    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2" podUID="bacfb6c5-09e6-4dfb-80cb-22336fdaac43" containerName="payload-processing" containerID="cri-o://36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f" gracePeriod=25
Nov 02 21:39:21.050232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.050146    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx" podUID="790d52a0-4dcc-41d4-954d-691075117128" containerName="payload-pre-processing" containerID="cri-o://fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869" gracePeriod=25
Nov 02 21:39:21.225485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.225461    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:39:21.261486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.261453    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-kube-api-access-xcld9\" (UniqueName: \"kubernetes.io/projected/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-kube-api-access-xcld9\") pod \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\" (UID: \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\") "
Nov 02 21:39:21.261683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.261664    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-plugins-config-volume\") pod \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\" (UID: \"bacfb6c5-09e6-4dfb-80cb-22336fdaac43\") "
Nov 02 21:39:21.262012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.261983    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-plugins-config-volume" pod "bacfb6c5-09e6-4dfb-80cb-22336fdaac43" (UID: "bacfb6c5-09e6-4dfb-80cb-22336fdaac43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:39:21.262206 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.262181    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:21.263530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.263508    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-kube-api-access-xcld9" pod "bacfb6c5-09e6-4dfb-80cb-22336fdaac43" (UID: "bacfb6c5-09e6-4dfb-80cb-22336fdaac43"). InnerVolumeSpecName "kube-api-access-xcld9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:39:21.317738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.317715    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:39:21.363474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.363447    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790d52a0-4dcc-41d4-954d-691075117128-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790d52a0-4dcc-41d4-954d-691075117128-plugins-config-volume\") pod \"790d52a0-4dcc-41d4-954d-691075117128\" (UID: \"790d52a0-4dcc-41d4-954d-691075117128\") "
Nov 02 21:39:21.363647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.363532    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790d52a0-4dcc-41d4-954d-691075117128-kube-api-access-9lht4\" (UniqueName: \"kubernetes.io/projected/790d52a0-4dcc-41d4-954d-691075117128-kube-api-access-9lht4\") pod \"790d52a0-4dcc-41d4-954d-691075117128\" (UID: \"790d52a0-4dcc-41d4-954d-691075117128\") "
Nov 02 21:39:21.363941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.363910    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790d52a0-4dcc-41d4-954d-691075117128-plugins-config-volume" pod "790d52a0-4dcc-41d4-954d-691075117128" (UID: "790d52a0-4dcc-41d4-954d-691075117128"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:39:21.364211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.364190    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcld9\" (UniqueName: \"kubernetes.io/projected/bacfb6c5-09e6-4dfb-80cb-22336fdaac43-kube-api-access-xcld9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:21.364798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.364217    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790d52a0-4dcc-41d4-954d-691075117128-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:21.365719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.365699    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790d52a0-4dcc-41d4-954d-691075117128-kube-api-access-9lht4" pod "790d52a0-4dcc-41d4-954d-691075117128" (UID: "790d52a0-4dcc-41d4-954d-691075117128"). InnerVolumeSpecName "kube-api-access-9lht4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:39:21.465898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.465818    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lht4\" (UniqueName: \"kubernetes.io/projected/790d52a0-4dcc-41d4-954d-691075117128-kube-api-access-9lht4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:39:21.472587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.472560    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:39:21.473039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.473016    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:39:21.526723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.526691    4528 generic.go:356] "Generic (PLEG): container finished" podID="bacfb6c5-09e6-4dfb-80cb-22336fdaac43" containerID="36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f" exitCode=0
Nov 02 21:39:21.526865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.526738    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"
Nov 02 21:39:21.526865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.526767    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2" event={"ID":"bacfb6c5-09e6-4dfb-80cb-22336fdaac43","Type":"ContainerDied","Data":"36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f"}
Nov 02 21:39:21.526865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.526811    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2" event={"ID":"bacfb6c5-09e6-4dfb-80cb-22336fdaac43","Type":"ContainerDied","Data":"78c6aa43833124cf203c5f25231a1a9dc532c6414b162c05c1988a8172fbf124"}
Nov 02 21:39:21.526865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.526833    4528 scope.go:122] "RemoveContainer" containerID="36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f"
Nov 02 21:39:21.528189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.528159    4528 generic.go:356] "Generic (PLEG): container finished" podID="790d52a0-4dcc-41d4-954d-691075117128" containerID="fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869" exitCode=0
Nov 02 21:39:21.528313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.528201    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"
Nov 02 21:39:21.528313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.528246    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx" event={"ID":"790d52a0-4dcc-41d4-954d-691075117128","Type":"ContainerDied","Data":"fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869"}
Nov 02 21:39:21.528313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.528296    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx" event={"ID":"790d52a0-4dcc-41d4-954d-691075117128","Type":"ContainerDied","Data":"19f8579041713b47e8db1a1a904808a521f8c9a813be3d34cbc384a42f73501d"}
Nov 02 21:39:21.537054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.537020    4528 scope.go:122] "RemoveContainer" containerID="36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f"
Nov 02 21:39:21.537414 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:39:21.537380    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f\": container with ID starting with 36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f not found: ID does not exist" containerID="36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f"
Nov 02 21:39:21.537482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.537417    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f"} err="failed to get container status \"36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f\": rpc error: code = NotFound desc = could not find container \"36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f\": container with ID starting with 36f407775795dd57a7e7ac93dd17b2e65523b343afc8d0a08497cb3e995d116f not found: ID does not exist"
Nov 02 21:39:21.537482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.537438    4528 scope.go:122] "RemoveContainer" containerID="fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869"
Nov 02 21:39:21.545765 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.545749    4528 scope.go:122] "RemoveContainer" containerID="fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869"
Nov 02 21:39:21.546006 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:39:21.545988    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869\": container with ID starting with fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869 not found: ID does not exist" containerID="fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869"
Nov 02 21:39:21.546080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.546012    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869"} err="failed to get container status \"fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869\": rpc error: code = NotFound desc = could not find container \"fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869\": container with ID starting with fe5364eadd286984110c0cce61dd42f2dd0ccbcf0526d9119411bc31f9628869 not found: ID does not exist"
Nov 02 21:39:21.708937 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.708900    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"]
Nov 02 21:39:21.711561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.711537    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-579bf7f87f-qmbdx"]
Nov 02 21:39:21.819038 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.819003    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"]
Nov 02 21:39:21.821936 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:21.821909    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-95f688dfc-qffm2"]
Nov 02 21:39:22.913637 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:22.913608    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790d52a0-4dcc-41d4-954d-691075117128" path="/var/lib/kubelet/pods/790d52a0-4dcc-41d4-954d-691075117128/volumes"
Nov 02 21:39:22.914197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:22.914166    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacfb6c5-09e6-4dfb-80cb-22336fdaac43" path="/var/lib/kubelet/pods/bacfb6c5-09e6-4dfb-80cb-22336fdaac43/volumes"
Nov 02 21:39:24.124369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.124337    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"]
Nov 02 21:39:24.124839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.124827    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bacfb6c5-09e6-4dfb-80cb-22336fdaac43" containerName="payload-processing"
Nov 02 21:39:24.124883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.124840    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacfb6c5-09e6-4dfb-80cb-22336fdaac43" containerName="payload-processing"
Nov 02 21:39:24.124883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.124858    4528 cpu_manager.go:414] "containerMap: removing container" podUID="790d52a0-4dcc-41d4-954d-691075117128" containerName="payload-pre-processing"
Nov 02 21:39:24.124883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.124863    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790d52a0-4dcc-41d4-954d-691075117128" containerName="payload-pre-processing"
Nov 02 21:39:24.125006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.124896    4528 cpu_manager.go:414] "containerMap: removing container" podUID="841b76a5-26ea-40f5-9e45-51d7609e670b" containerName="revoke-keys"
Nov 02 21:39:24.125006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.124908    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841b76a5-26ea-40f5-9e45-51d7609e670b" containerName="revoke-keys"
Nov 02 21:39:24.125006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.125001    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="841b76a5-26ea-40f5-9e45-51d7609e670b" containerName="revoke-keys"
Nov 02 21:39:24.125103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.125010    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacfb6c5-09e6-4dfb-80cb-22336fdaac43" containerName="payload-processing"
Nov 02 21:39:24.125103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.125018    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="790d52a0-4dcc-41d4-954d-691075117128" containerName="payload-pre-processing"
Nov 02 21:39:24.129868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.129847    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.185249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.184471    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4453bd-openshift-default-dockercfg-zkpv2\""
Nov 02 21:39:24.249144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.196095    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-socket\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.292305 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.292267    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-token\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.292558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.292535    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istiod-ca-cert\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.292701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.292684    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7lh\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-kube-api-access-7d7lh\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.292910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.292881    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-envoy\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.293099 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.293079    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-podinfo\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.293292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.293270    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-data\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.293406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.293390    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-certs\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.308139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.293653    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-credential-socket\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.316213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.316175    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"]
Nov 02 21:39:24.409721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.409636    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-podinfo\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.409721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.409695    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-data\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.409988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.409723    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-certs\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.409988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.409864    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-credential-socket\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.409988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.409912    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-socket\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.409988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.409971    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-token\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.410256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.410000    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istiod-ca-cert\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.410256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.410038    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7lh\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-kube-api-access-7d7lh\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.410256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.410189    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-envoy\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.410414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.410253    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-certs\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.410414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.410304    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-credential-socket\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.410560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.410539    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-data\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.410777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.410756    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istiod-ca-cert\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.410813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.410787    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-socket\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.412404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.412383    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-podinfo\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.412679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.412664    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-envoy\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.467362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.467325    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-token\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.467511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.467443    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7lh\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-kube-api-access-7d7lh\") pod \"e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") " pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:24.742934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:24.742845    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:25.162093 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:39:25.162057    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7af38e3_4322_46b6_9eb6_638c9a3e5be0.slice/crio-bf95a35340c2fc4668be1fd3dc8a2089f30d460bbb8a439b68bc5a069bc96cb8 WatchSource:0}: Error finding container bf95a35340c2fc4668be1fd3dc8a2089f30d460bbb8a439b68bc5a069bc96cb8: Status 404 returned error can't find the container with id bf95a35340c2fc4668be1fd3dc8a2089f30d460bbb8a439b68bc5a069bc96cb8
Nov 02 21:39:25.164623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:25.164601    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"]
Nov 02 21:39:25.164716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:25.164672    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:39:25.164759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:25.164734    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:39:25.164792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:25.164785    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:39:25.556054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:25.556014    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" event={"ID":"a7af38e3-4322-46b6-9eb6-638c9a3e5be0","Type":"ContainerStarted","Data":"f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8"}
Nov 02 21:39:25.556263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:25.556080    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" event={"ID":"a7af38e3-4322-46b6-9eb6-638c9a3e5be0","Type":"ContainerStarted","Data":"bf95a35340c2fc4668be1fd3dc8a2089f30d460bbb8a439b68bc5a069bc96cb8"}
Nov 02 21:39:25.674551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:25.674504    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" podStartSLOduration=1.674489501 podStartE2EDuration="1.674489501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:25.672590724 +0000 UTC m=+1852.936127041" watchObservedRunningTime="2026-11-02 21:39:25.674489501 +0000 UTC m=+1852.938025819"
Nov 02 21:39:25.743391 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:25.743361    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:26.748865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:26.748838    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:27.567912 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:27.567881    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:27.568838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:27.568817    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:39:34.184151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.175541    4528 scope.go:122] "RemoveContainer" containerID="494c51b1d0b74c71c2572218505d424d48d8d91986b857e79687588e94793c15"
Nov 02 21:39:34.242217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.242171    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"]
Nov 02 21:39:34.247486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.247467    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:34.297172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.297140    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"]
Nov 02 21:39:34.344628 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.344588    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4453bd\""
Nov 02 21:39:34.344787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.344651    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4453bd-dockercfg-w7shl\""
Nov 02 21:39:34.352358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.352327    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10d1567-6ff3-4257-9635-55320b8f3b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x\" (UID: \"b10d1567-6ff3-4257-9635-55320b8f3b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:34.352487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.352473    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4vm\" (UniqueName: \"kubernetes.io/projected/b10d1567-6ff3-4257-9635-55320b8f3b2f-kube-api-access-tn4vm\") pod \"payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x\" (UID: \"b10d1567-6ff3-4257-9635-55320b8f3b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:34.401189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.401152    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"]
Nov 02 21:39:34.432132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.426297    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:34.456937 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.456848    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10d1567-6ff3-4257-9635-55320b8f3b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x\" (UID: \"b10d1567-6ff3-4257-9635-55320b8f3b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:34.471886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.471731    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10d1567-6ff3-4257-9635-55320b8f3b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x\" (UID: \"b10d1567-6ff3-4257-9635-55320b8f3b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:34.472253 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.472226    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4vm\" (UniqueName: \"kubernetes.io/projected/b10d1567-6ff3-4257-9635-55320b8f3b2f-kube-api-access-tn4vm\") pod \"payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x\" (UID: \"b10d1567-6ff3-4257-9635-55320b8f3b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:34.483273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.474265    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"]
Nov 02 21:39:34.531706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.531679    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4vm\" (UniqueName: \"kubernetes.io/projected/b10d1567-6ff3-4257-9635-55320b8f3b2f-kube-api-access-tn4vm\") pod \"payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x\" (UID: \"b10d1567-6ff3-4257-9635-55320b8f3b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:34.564874 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.564848    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:34.573393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.573363    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7kzl\" (UniqueName: \"kubernetes.io/projected/01170016-26dc-45ec-9397-4c4483b38d39-kube-api-access-q7kzl\") pod \"payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f\" (UID: \"01170016-26dc-45ec-9397-4c4483b38d39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:34.573575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.573555    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01170016-26dc-45ec-9397-4c4483b38d39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f\" (UID: \"01170016-26dc-45ec-9397-4c4483b38d39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:34.675237 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.675189    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7kzl\" (UniqueName: \"kubernetes.io/projected/01170016-26dc-45ec-9397-4c4483b38d39-kube-api-access-q7kzl\") pod \"payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f\" (UID: \"01170016-26dc-45ec-9397-4c4483b38d39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:34.675438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.675415    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01170016-26dc-45ec-9397-4c4483b38d39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f\" (UID: \"01170016-26dc-45ec-9397-4c4483b38d39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:34.676148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.676127    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01170016-26dc-45ec-9397-4c4483b38d39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f\" (UID: \"01170016-26dc-45ec-9397-4c4483b38d39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:34.733018 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.732932    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7kzl\" (UniqueName: \"kubernetes.io/projected/01170016-26dc-45ec-9397-4c4483b38d39-kube-api-access-q7kzl\") pod \"payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f\" (UID: \"01170016-26dc-45ec-9397-4c4483b38d39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:34.821414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.821384    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:34.940438 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:39:34.940404    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10d1567_6ff3_4257_9635_55320b8f3b2f.slice/crio-c6673bd4f29dfbaeb94d8a00f33c71259eefa04abb54419fd8cf996c9d4a0671 WatchSource:0}: Error finding container c6673bd4f29dfbaeb94d8a00f33c71259eefa04abb54419fd8cf996c9d4a0671: Status 404 returned error can't find the container with id c6673bd4f29dfbaeb94d8a00f33c71259eefa04abb54419fd8cf996c9d4a0671
Nov 02 21:39:34.943588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:34.943562    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"]
Nov 02 21:39:35.220931 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:35.218732    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"]
Nov 02 21:39:35.617799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:35.617762    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f" event={"ID":"01170016-26dc-45ec-9397-4c4483b38d39","Type":"ContainerStarted","Data":"74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603"}
Nov 02 21:39:35.618086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:35.618064    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:35.618167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:35.618148    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f" event={"ID":"01170016-26dc-45ec-9397-4c4483b38d39","Type":"ContainerStarted","Data":"bd6e1b4d7b2d20c237f4813354be2d7234cf190b42712dfa34cfc1a75c029e3e"}
Nov 02 21:39:35.619466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:35.619441    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x" event={"ID":"b10d1567-6ff3-4257-9635-55320b8f3b2f","Type":"ContainerStarted","Data":"bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc"}
Nov 02 21:39:35.619559 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:35.619483    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x" event={"ID":"b10d1567-6ff3-4257-9635-55320b8f3b2f","Type":"ContainerStarted","Data":"c6673bd4f29dfbaeb94d8a00f33c71259eefa04abb54419fd8cf996c9d4a0671"}
Nov 02 21:39:35.619599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:35.619568    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:35.736059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:35.735904    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f" podStartSLOduration=1.735879001 podStartE2EDuration="1.735879001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:35.73168339 +0000 UTC m=+1862.995219719" watchObservedRunningTime="2026-11-02 21:39:35.735879001 +0000 UTC m=+1862.999415338"
Nov 02 21:39:35.849784 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:35.849722    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x" podStartSLOduration=1.849701153 podStartE2EDuration="1.849701153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:35.846683309 +0000 UTC m=+1863.110219626" watchObservedRunningTime="2026-11-02 21:39:35.849701153 +0000 UTC m=+1863.113237472"
Nov 02 21:39:36.744149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:36.744105    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"]
Nov 02 21:39:36.748181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:36.748160    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:36.801409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:36.801380    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"]
Nov 02 21:39:36.847214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:36.847187    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-15253e\""
Nov 02 21:39:36.847543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:36.847520    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-15253e-dockercfg-xfnpt\""
Nov 02 21:39:36.920176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:36.920147    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2pn\" (UniqueName: \"kubernetes.io/projected/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-kube-api-access-pq2pn\") pod \"payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j\" (UID: \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:36.920337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:36.920212    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j\" (UID: \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:37.021382 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.021354    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2pn\" (UniqueName: \"kubernetes.io/projected/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-kube-api-access-pq2pn\") pod \"payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j\" (UID: \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:37.021601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.021581    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j\" (UID: \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:37.022307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.022279    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j\" (UID: \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:37.078380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.078355    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2pn\" (UniqueName: \"kubernetes.io/projected/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-kube-api-access-pq2pn\") pod \"payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j\" (UID: \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:37.362324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.362244    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:37.489719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.489683    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"]
Nov 02 21:39:37.495612 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.495590    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:37.545752 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.545712    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"]
Nov 02 21:39:37.629864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.629797    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/be7bf237-e30a-4102-b487-9f5486b259fa-kube-api-access-f87mt\") pod \"payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm\" (UID: \"be7bf237-e30a-4102-b487-9f5486b259fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:37.630017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.629870    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7bf237-e30a-4102-b487-9f5486b259fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm\" (UID: \"be7bf237-e30a-4102-b487-9f5486b259fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:37.731469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.731435    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7bf237-e30a-4102-b487-9f5486b259fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm\" (UID: \"be7bf237-e30a-4102-b487-9f5486b259fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:37.731644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.731595    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/be7bf237-e30a-4102-b487-9f5486b259fa-kube-api-access-f87mt\") pod \"payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm\" (UID: \"be7bf237-e30a-4102-b487-9f5486b259fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:37.732162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.732144    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7bf237-e30a-4102-b487-9f5486b259fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm\" (UID: \"be7bf237-e30a-4102-b487-9f5486b259fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:37.789637 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.789600    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/be7bf237-e30a-4102-b487-9f5486b259fa-kube-api-access-f87mt\") pod \"payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm\" (UID: \"be7bf237-e30a-4102-b487-9f5486b259fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:37.802089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.802060    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"]
Nov 02 21:39:37.809885 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:37.809864    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:38.217143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:38.211769    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"]
Nov 02 21:39:38.217143 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:39:38.213242    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7bf237_e30a_4102_b487_9f5486b259fa.slice/crio-81e86ff2c67e4be6d66bae60ce7fb3e98ac579e01c1afa20681c34bda9f38ecf WatchSource:0}: Error finding container 81e86ff2c67e4be6d66bae60ce7fb3e98ac579e01c1afa20681c34bda9f38ecf: Status 404 returned error can't find the container with id 81e86ff2c67e4be6d66bae60ce7fb3e98ac579e01c1afa20681c34bda9f38ecf
Nov 02 21:39:38.640649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:38.640612    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm" event={"ID":"be7bf237-e30a-4102-b487-9f5486b259fa","Type":"ContainerStarted","Data":"8742786dee8113cd6a279824949fd88f18ee8a5b7652f2d9a592489e1ac4027a"}
Nov 02 21:39:38.640834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:38.640670    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm" event={"ID":"be7bf237-e30a-4102-b487-9f5486b259fa","Type":"ContainerStarted","Data":"81e86ff2c67e4be6d66bae60ce7fb3e98ac579e01c1afa20681c34bda9f38ecf"}
Nov 02 21:39:38.640834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:38.640697    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:38.642332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:38.642306    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j" event={"ID":"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c","Type":"ContainerStarted","Data":"4685c5f97c736d488dd8b003fbb8cf66ebcda255ebc36534bff98b47942ff120"}
Nov 02 21:39:38.642466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:38.642349    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j" event={"ID":"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c","Type":"ContainerStarted","Data":"a717cb920b4381840521e7c8d51646fb13944be242756343c04ed6c16276c31c"}
Nov 02 21:39:38.642466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:38.642385    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:38.758100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:38.758053    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm" podStartSLOduration=1.758040322 podStartE2EDuration="1.758040322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:38.75532318 +0000 UTC m=+1866.018859498" watchObservedRunningTime="2026-11-02 21:39:38.758040322 +0000 UTC m=+1866.021576640"
Nov 02 21:39:38.878754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:38.872884    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j" podStartSLOduration=2.872865095 podStartE2EDuration="2.872865095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:38.869031686 +0000 UTC m=+1866.132568006" watchObservedRunningTime="2026-11-02 21:39:38.872865095 +0000 UTC m=+1866.136401412"
Nov 02 21:39:44.047560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:44.047520    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"]
Nov 02 21:39:44.065495 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:44.065468    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2a5ff8-8jqfv"]
Nov 02 21:39:44.114059 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:39:44.114023    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:39:44.531616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:44.531580    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:39:44.913725 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:44.913647    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e35bf9f3-4deb-417e-a3cb-7facb777aa87" path="/var/lib/kubelet/pods/e35bf9f3-4deb-417e-a3cb-7facb777aa87/volumes"
Nov 02 21:39:46.629236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:46.629209    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:39:46.629685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:46.629276    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:39:49.650572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:49.650540    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:39:49.651061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:49.650624    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:39:54.760739 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:54.760710    4528 generic.go:356] "Generic (PLEG): container finished" podID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerID="4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c" exitCode=0
Nov 02 21:39:54.761302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:54.760789    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" event={"ID":"3734a586-cc32-4840-af2a-0d7d4b63bc46","Type":"ContainerDied","Data":"4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c"}
Nov 02 21:39:54.761388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:54.761355    4528 scope.go:122] "RemoveContainer" containerID="4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c"
Nov 02 21:39:54.762760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:54.762733    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:39:54.762848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:54.762800    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:39:54.762848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:54.762831    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:39:55.781772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:55.781733    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" event={"ID":"3734a586-cc32-4840-af2a-0d7d4b63bc46","Type":"ContainerStarted","Data":"1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a"}
Nov 02 21:39:56.028719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.027330    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"]
Nov 02 21:39:56.034609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.034536    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.132131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.131034    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"]
Nov 02 21:39:56.181161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.173475    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6a9abc-openshift-default-dockercfg-jmj9c\""
Nov 02 21:39:56.210869 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.210835    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-token\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.215265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.215235    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-podinfo\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.239132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.215544    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-envoy\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.269756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.261943    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-credential-socket\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.269756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.262058    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cjxp\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-kube-api-access-8cjxp\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.269756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.262162    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-certs\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.269756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.262240    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-data\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.269756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.262274    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-socket\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.269756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.262328    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istiod-ca-cert\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363349    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cjxp\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-kube-api-access-8cjxp\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363409    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-certs\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363441    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-data\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363461    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-socket\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363491    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istiod-ca-cert\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363542    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-token\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363559    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-podinfo\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363608    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-envoy\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363804 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363635    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-credential-socket\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.363990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.363974    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-credential-socket\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.364402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.364386    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-certs\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.364660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.364542    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-data\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.364782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.364689    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-socket\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.365167 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.365145    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istiod-ca-cert\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.369248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.369222    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-envoy\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.369935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.369915    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-podinfo\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.428457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.427176    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cjxp\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-kube-api-access-8cjxp\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.438074 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.438039    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-token\") pod \"e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") " pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:56.574365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.574333    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:39:56.574566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.574540    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 21:39:56.574641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.574599    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 21:39:56.662442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:56.662361    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:57.130240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:57.130213    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"]
Nov 02 21:39:57.130575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:57.130407    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:39:57.130575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:57.130461    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:39:57.130575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:57.130489    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:39:57.573781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:57.573747    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 21:39:57.573983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:57.573799    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 21:39:57.797913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:57.797866    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" event={"ID":"e733fbf4-28fd-4a07-8357-a372bc92d8b3","Type":"ContainerStarted","Data":"6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f"}
Nov 02 21:39:57.798105 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:57.797972    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" event={"ID":"e733fbf4-28fd-4a07-8357-a372bc92d8b3","Type":"ContainerStarted","Data":"6d420d7a161add09f2aa9bcbf875ba7485ed142f4e7d731279b6e91de9123c4c"}
Nov 02 21:39:57.927085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:57.926967    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" podStartSLOduration=2.926952044 podStartE2EDuration="2.926952044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:57.926374554 +0000 UTC m=+1885.189910885" watchObservedRunningTime="2026-11-02 21:39:57.926952044 +0000 UTC m=+1885.190488360"
Nov 02 21:39:58.574162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:58.574098    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 21:39:58.574614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:58.574195    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 21:39:58.662867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:58.662835    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:58.668238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:58.668217    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:58.826428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:58.826340    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:58.827456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:58.827434    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:39:59.574187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:59.574143    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 21:39:59.574563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:39:59.574207    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 21:40:00.573821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:00.573788    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 21:40:00.573997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:00.573869    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 21:40:01.577775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:01.577746    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:40:01.578274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:01.578014    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:40:01.578772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:01.578754    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:40:04.599921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.599880    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"]
Nov 02 21:40:04.604156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.604138    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:04.707131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.688891    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"]
Nov 02 21:40:04.708256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.708221    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clgzc\" (UniqueName: \"kubernetes.io/projected/ccd3ccbb-c199-4540-8e31-0e7c53f44027-kube-api-access-clgzc\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55\" (UID: \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:04.708500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.708479    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd3ccbb-c199-4540-8e31-0e7c53f44027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55\" (UID: \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:04.718403 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.718376    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a9abc\""
Nov 02 21:40:04.718708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.718691    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a9abc-dockercfg-6ffgb\""
Nov 02 21:40:04.762712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.762663    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"]
Nov 02 21:40:04.771226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.770585    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:04.811066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.810285    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbde66-94cd-4205-b99a-197f1b14c84d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs\" (UID: \"b0cbde66-94cd-4205-b99a-197f1b14c84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:04.811066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.810354    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clgzc\" (UniqueName: \"kubernetes.io/projected/ccd3ccbb-c199-4540-8e31-0e7c53f44027-kube-api-access-clgzc\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55\" (UID: \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:04.811066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.810435    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd3ccbb-c199-4540-8e31-0e7c53f44027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55\" (UID: \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:04.811066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.810480    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ccbh\" (UniqueName: \"kubernetes.io/projected/b0cbde66-94cd-4205-b99a-197f1b14c84d-kube-api-access-5ccbh\") pod \"payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs\" (UID: \"b0cbde66-94cd-4205-b99a-197f1b14c84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:04.811436 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.811098    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd3ccbb-c199-4540-8e31-0e7c53f44027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55\" (UID: \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:04.826463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.826430    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"]
Nov 02 21:40:04.867812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.867740    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clgzc\" (UniqueName: \"kubernetes.io/projected/ccd3ccbb-c199-4540-8e31-0e7c53f44027-kube-api-access-clgzc\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55\" (UID: \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:04.911059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.911031    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbde66-94cd-4205-b99a-197f1b14c84d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs\" (UID: \"b0cbde66-94cd-4205-b99a-197f1b14c84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:04.911243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.911096    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ccbh\" (UniqueName: \"kubernetes.io/projected/b0cbde66-94cd-4205-b99a-197f1b14c84d-kube-api-access-5ccbh\") pod \"payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs\" (UID: \"b0cbde66-94cd-4205-b99a-197f1b14c84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:04.911698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.911673    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbde66-94cd-4205-b99a-197f1b14c84d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs\" (UID: \"b0cbde66-94cd-4205-b99a-197f1b14c84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:04.917970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.917952    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:04.970402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:04.968686    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ccbh\" (UniqueName: \"kubernetes.io/projected/b0cbde66-94cd-4205-b99a-197f1b14c84d-kube-api-access-5ccbh\") pod \"payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs\" (UID: \"b0cbde66-94cd-4205-b99a-197f1b14c84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:05.101271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.101240    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:05.287806 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:05.287776    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd3ccbb_c199_4540_8e31_0e7c53f44027.slice/crio-fb8ede247c3943968e8889dc158500b657bc1a962793c24a75ea27412bb454f0 WatchSource:0}: Error finding container fb8ede247c3943968e8889dc158500b657bc1a962793c24a75ea27412bb454f0: Status 404 returned error can't find the container with id fb8ede247c3943968e8889dc158500b657bc1a962793c24a75ea27412bb454f0
Nov 02 21:40:05.290404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.290381    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"]
Nov 02 21:40:05.468095 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:05.468063    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cbde66_94cd_4205_b99a_197f1b14c84d.slice/crio-c0147f13da40d7afee05b22c44de2c1451d8c05c522051bb3d0fde166dc646e8 WatchSource:0}: Error finding container c0147f13da40d7afee05b22c44de2c1451d8c05c522051bb3d0fde166dc646e8: Status 404 returned error can't find the container with id c0147f13da40d7afee05b22c44de2c1451d8c05c522051bb3d0fde166dc646e8
Nov 02 21:40:05.471100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.471046    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"]
Nov 02 21:40:05.869942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.869905    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs" event={"ID":"b0cbde66-94cd-4205-b99a-197f1b14c84d","Type":"ContainerStarted","Data":"8d9f0e9b0e21406a08e56b2eb674623949528d344db8940cd8f6d7491f4b0769"}
Nov 02 21:40:05.870380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.869978    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:05.870380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.870001    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs" event={"ID":"b0cbde66-94cd-4205-b99a-197f1b14c84d","Type":"ContainerStarted","Data":"c0147f13da40d7afee05b22c44de2c1451d8c05c522051bb3d0fde166dc646e8"}
Nov 02 21:40:05.871556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.871530    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55" event={"ID":"ccd3ccbb-c199-4540-8e31-0e7c53f44027","Type":"ContainerStarted","Data":"3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24"}
Nov 02 21:40:05.871689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.871588    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:05.871689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.871614    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55" event={"ID":"ccd3ccbb-c199-4540-8e31-0e7c53f44027","Type":"ContainerStarted","Data":"fb8ede247c3943968e8889dc158500b657bc1a962793c24a75ea27412bb454f0"}
Nov 02 21:40:05.985488 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:05.985437    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs" podStartSLOduration=1.985420682 podStartE2EDuration="1.985420682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:05.98466075 +0000 UTC m=+1893.248197066" watchObservedRunningTime="2026-11-02 21:40:05.985420682 +0000 UTC m=+1893.248956999"
Nov 02 21:40:06.099773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:06.099719    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55" podStartSLOduration=2.099700697 podStartE2EDuration="2.099700697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:06.097385895 +0000 UTC m=+1893.360922213" watchObservedRunningTime="2026-11-02 21:40:06.099700697 +0000 UTC m=+1893.363237013"
Nov 02 21:40:08.108218 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.108182    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"]
Nov 02 21:40:08.162138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.159382    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70efbbab-r78zv"]
Nov 02 21:40:08.324139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.324080    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"]
Nov 02 21:40:08.324398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.324373    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy" containerID="cri-o://ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7" gracePeriod=30
Nov 02 21:40:08.364217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.364138    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"]
Nov 02 21:40:08.364370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.364348    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy" containerID="cri-o://7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca" gracePeriod=30
Nov 02 21:40:08.369346 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.369321    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"]
Nov 02 21:40:08.369522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.369501    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy" containerID="cri-o://1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601" gracePeriod=30
Nov 02 21:40:08.522980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.522942    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:40:08.523278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.523247    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:40:08.863563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.863525    4528 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:40:08.863709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.863580    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:40:08.913636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:08.913601    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841b76a5-26ea-40f5-9e45-51d7609e670b" path="/var/lib/kubelet/pods/841b76a5-26ea-40f5-9e45-51d7609e670b/volumes"
Nov 02 21:40:13.657798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.657766    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:40:13.681528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.681505    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:40:13.709221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.709197    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:40:13.758282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758249    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-credential-socket\") pod \"ebe5a031-de99-4fb9-b118-68ba39cf459b\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") "
Nov 02 21:40:13.758424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758285    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-data\") pod \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") "
Nov 02 21:40:13.758424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758323    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-kube-api-access-2kpks\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-kube-api-access-2kpks\") pod \"48771cb7-31e2-4192-99a2-e0e36f6d8737\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") "
Nov 02 21:40:13.758424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758375    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-socket\") pod \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") "
Nov 02 21:40:13.758424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758410    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-certs\") pod \"ebe5a031-de99-4fb9-b118-68ba39cf459b\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") "
Nov 02 21:40:13.758653 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758437    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-token\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-token\") pod \"ebe5a031-de99-4fb9-b118-68ba39cf459b\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") "
Nov 02 21:40:13.758653 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758503    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-envoy\") pod \"48771cb7-31e2-4192-99a2-e0e36f6d8737\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") "
Nov 02 21:40:13.758653 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758526    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-data" pod "2ae9bbaf-405b-43a8-b353-73a2265a1ad3" (UID: "2ae9bbaf-405b-43a8-b353-73a2265a1ad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.758653 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758534    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-credential-socket" pod "ebe5a031-de99-4fb9-b118-68ba39cf459b" (UID: "ebe5a031-de99-4fb9-b118-68ba39cf459b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.758653 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758539    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-podinfo\") pod \"ebe5a031-de99-4fb9-b118-68ba39cf459b\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") "
Nov 02 21:40:13.758921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758657    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-socket" pod "2ae9bbaf-405b-43a8-b353-73a2265a1ad3" (UID: "2ae9bbaf-405b-43a8-b353-73a2265a1ad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.758921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758688    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-certs\") pod \"48771cb7-31e2-4192-99a2-e0e36f6d8737\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") "
Nov 02 21:40:13.758921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758728    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-credential-socket\") pod \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") "
Nov 02 21:40:13.758921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758688    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-certs" pod "ebe5a031-de99-4fb9-b118-68ba39cf459b" (UID: "ebe5a031-de99-4fb9-b118-68ba39cf459b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.758921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758760    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48771cb7-31e2-4192-99a2-e0e36f6d8737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48771cb7-31e2-4192-99a2-e0e36f6d8737-istiod-ca-cert\") pod \"48771cb7-31e2-4192-99a2-e0e36f6d8737\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") "
Nov 02 21:40:13.758921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758801    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-envoy\") pod \"ebe5a031-de99-4fb9-b118-68ba39cf459b\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") "
Nov 02 21:40:13.758921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758846    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-data\") pod \"ebe5a031-de99-4fb9-b118-68ba39cf459b\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") "
Nov 02 21:40:13.758921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758874    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-certs" pod "48771cb7-31e2-4192-99a2-e0e36f6d8737" (UID: "48771cb7-31e2-4192-99a2-e0e36f6d8737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.758921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758878    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-socket\") pod \"ebe5a031-de99-4fb9-b118-68ba39cf459b\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758967    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-kube-api-access-z596c\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-kube-api-access-z596c\") pod \"ebe5a031-de99-4fb9-b118-68ba39cf459b\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.758980    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-credential-socket" pod "2ae9bbaf-405b-43a8-b353-73a2265a1ad3" (UID: "2ae9bbaf-405b-43a8-b353-73a2265a1ad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759013    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-token\") pod \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759034    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-socket" pod "ebe5a031-de99-4fb9-b118-68ba39cf459b" (UID: "ebe5a031-de99-4fb9-b118-68ba39cf459b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759057    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-certs\") pod \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759139    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-podinfo\") pod \"48771cb7-31e2-4192-99a2-e0e36f6d8737\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759184    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe5a031-de99-4fb9-b118-68ba39cf459b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe5a031-de99-4fb9-b118-68ba39cf459b-istiod-ca-cert\") pod \"ebe5a031-de99-4fb9-b118-68ba39cf459b\" (UID: \"ebe5a031-de99-4fb9-b118-68ba39cf459b\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759216    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-envoy\") pod \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759235    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-data" pod "ebe5a031-de99-4fb9-b118-68ba39cf459b" (UID: "ebe5a031-de99-4fb9-b118-68ba39cf459b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759245    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istiod-ca-cert\") pod \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759306    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-data\") pod \"48771cb7-31e2-4192-99a2-e0e36f6d8737\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759336    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-credential-socket\") pod \"48771cb7-31e2-4192-99a2-e0e36f6d8737\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759385    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-socket\") pod \"48771cb7-31e2-4192-99a2-e0e36f6d8737\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") "
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759408    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48771cb7-31e2-4192-99a2-e0e36f6d8737-istiod-ca-cert" pod "48771cb7-31e2-4192-99a2-e0e36f6d8737" (UID: "48771cb7-31e2-4192-99a2-e0e36f6d8737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:40:13.759423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759419    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-kube-api-access-2f5r8\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-kube-api-access-2f5r8\") pod \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") "
Nov 02 21:40:13.760264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759493    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-token\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-token\") pod \"48771cb7-31e2-4192-99a2-e0e36f6d8737\" (UID: \"48771cb7-31e2-4192-99a2-e0e36f6d8737\") "
Nov 02 21:40:13.760264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.759539    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-podinfo\") pod \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\" (UID: \"2ae9bbaf-405b-43a8-b353-73a2265a1ad3\") "
Nov 02 21:40:13.760264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.760203    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.760264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.760226    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.760264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.760242    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48771cb7-31e2-4192-99a2-e0e36f6d8737-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.760264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.760261    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.760595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.760275    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.760595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.760289    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.760595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.760304    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.760595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.760317    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.760595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.760333    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.761295 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.761270    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istiod-ca-cert" pod "2ae9bbaf-405b-43a8-b353-73a2265a1ad3" (UID: "2ae9bbaf-405b-43a8-b353-73a2265a1ad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:40:13.761438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.761324    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe5a031-de99-4fb9-b118-68ba39cf459b-istiod-ca-cert" pod "ebe5a031-de99-4fb9-b118-68ba39cf459b" (UID: "ebe5a031-de99-4fb9-b118-68ba39cf459b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:40:13.761576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.761554    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-kube-api-access-2kpks" pod "48771cb7-31e2-4192-99a2-e0e36f6d8737" (UID: "48771cb7-31e2-4192-99a2-e0e36f6d8737"). InnerVolumeSpecName "kube-api-access-2kpks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:13.761840 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.761789    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-socket" pod "48771cb7-31e2-4192-99a2-e0e36f6d8737" (UID: "48771cb7-31e2-4192-99a2-e0e36f6d8737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.762016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.761995    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-data" pod "48771cb7-31e2-4192-99a2-e0e36f6d8737" (UID: "48771cb7-31e2-4192-99a2-e0e36f6d8737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.762187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.762151    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-certs" pod "2ae9bbaf-405b-43a8-b353-73a2265a1ad3" (UID: "2ae9bbaf-405b-43a8-b353-73a2265a1ad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.762274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.762226    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-credential-socket" pod "48771cb7-31e2-4192-99a2-e0e36f6d8737" (UID: "48771cb7-31e2-4192-99a2-e0e36f6d8737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.762522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.762487    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-envoy" pod "48771cb7-31e2-4192-99a2-e0e36f6d8737" (UID: "48771cb7-31e2-4192-99a2-e0e36f6d8737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.762662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.762549    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-podinfo" pod "ebe5a031-de99-4fb9-b118-68ba39cf459b" (UID: "ebe5a031-de99-4fb9-b118-68ba39cf459b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:40:13.763473 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.763447    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-kube-api-access-z596c" pod "ebe5a031-de99-4fb9-b118-68ba39cf459b" (UID: "ebe5a031-de99-4fb9-b118-68ba39cf459b"). InnerVolumeSpecName "kube-api-access-z596c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:13.763573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.763545    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-envoy" pod "ebe5a031-de99-4fb9-b118-68ba39cf459b" (UID: "ebe5a031-de99-4fb9-b118-68ba39cf459b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.763728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.763705    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-token" pod "2ae9bbaf-405b-43a8-b353-73a2265a1ad3" (UID: "2ae9bbaf-405b-43a8-b353-73a2265a1ad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:13.764019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.763987    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-token" pod "48771cb7-31e2-4192-99a2-e0e36f6d8737" (UID: "48771cb7-31e2-4192-99a2-e0e36f6d8737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:13.764148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.764101    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-envoy" pod "2ae9bbaf-405b-43a8-b353-73a2265a1ad3" (UID: "2ae9bbaf-405b-43a8-b353-73a2265a1ad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:40:13.764211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.764179    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-kube-api-access-2f5r8" pod "2ae9bbaf-405b-43a8-b353-73a2265a1ad3" (UID: "2ae9bbaf-405b-43a8-b353-73a2265a1ad3"). InnerVolumeSpecName "kube-api-access-2f5r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:13.764275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.764261    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-podinfo" pod "2ae9bbaf-405b-43a8-b353-73a2265a1ad3" (UID: "2ae9bbaf-405b-43a8-b353-73a2265a1ad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:40:13.764467 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.764448    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-podinfo" pod "48771cb7-31e2-4192-99a2-e0e36f6d8737" (UID: "48771cb7-31e2-4192-99a2-e0e36f6d8737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:40:13.764625 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.764607    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-token" pod "ebe5a031-de99-4fb9-b118-68ba39cf459b" (UID: "ebe5a031-de99-4fb9-b118-68ba39cf459b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:13.765604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.765583    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"]
Nov 02 21:40:13.766234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766209    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy"
Nov 02 21:40:13.766234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766227    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy"
Nov 02 21:40:13.766369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766248    4528 cpu_manager.go:414] "containerMap: removing container" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy"
Nov 02 21:40:13.766369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766257    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy"
Nov 02 21:40:13.766369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766287    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy"
Nov 02 21:40:13.766369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766296    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy"
Nov 02 21:40:13.766369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766322    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy"
Nov 02 21:40:13.766369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766329    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy"
Nov 02 21:40:13.766651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766468    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy"
Nov 02 21:40:13.766651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766486    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerName="istio-proxy"
Nov 02 21:40:13.766651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766499    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerName="istio-proxy"
Nov 02 21:40:13.766651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.766513    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerName="istio-proxy"
Nov 02 21:40:13.771145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.771129    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:13.801102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.801075    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"]
Nov 02 21:40:13.826609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.826583    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7d3617-dockercfg-z4rtv\""
Nov 02 21:40:13.826752 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.826731    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7d3617\""
Nov 02 21:40:13.861929 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.861897    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xkh\" (UniqueName: \"kubernetes.io/projected/3aac0858-7c8f-4144-b5b5-ed9389162e81-kube-api-access-75xkh\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq\" (UID: \"3aac0858-7c8f-4144-b5b5-ed9389162e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:13.862159 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862142    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aac0858-7c8f-4144-b5b5-ed9389162e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq\" (UID: \"3aac0858-7c8f-4144-b5b5-ed9389162e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:13.862305 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862291    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z596c\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-kube-api-access-z596c\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862312    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862326    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862339    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862354    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe5a031-de99-4fb9-b118-68ba39cf459b-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862367    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862380    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862396    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862410    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862423    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862436    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f5r8\" (UniqueName: \"kubernetes.io/projected/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-kube-api-access-2f5r8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862451    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862464    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae9bbaf-405b-43a8-b353-73a2265a1ad3-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862479    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kpks\" (UniqueName: \"kubernetes.io/projected/48771cb7-31e2-4192-99a2-e0e36f6d8737-kube-api-access-2kpks\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862492    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862504    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48771cb7-31e2-4192-99a2-e0e36f6d8737-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862517    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.862610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.862531    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe5a031-de99-4fb9-b118-68ba39cf459b-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:13.926379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.926299    4528 generic.go:356] "Generic (PLEG): container finished" podID="ebe5a031-de99-4fb9-b118-68ba39cf459b" containerID="ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7" exitCode=0
Nov 02 21:40:13.926379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.926349    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"
Nov 02 21:40:13.926379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.926363    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" event={"ID":"ebe5a031-de99-4fb9-b118-68ba39cf459b","Type":"ContainerDied","Data":"ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7"}
Nov 02 21:40:13.926616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.926419    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4" event={"ID":"ebe5a031-de99-4fb9-b118-68ba39cf459b","Type":"ContainerDied","Data":"2ba637b610e12b6d1addb6ad2e145dd646ea7c10cdf79dcf8558d4fd3e8cc468"}
Nov 02 21:40:13.926616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.926450    4528 scope.go:122] "RemoveContainer" containerID="ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7"
Nov 02 21:40:13.927914 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.927895    4528 generic.go:356] "Generic (PLEG): container finished" podID="48771cb7-31e2-4192-99a2-e0e36f6d8737" containerID="1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601" exitCode=0
Nov 02 21:40:13.928016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.927944    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" event={"ID":"48771cb7-31e2-4192-99a2-e0e36f6d8737","Type":"ContainerDied","Data":"1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601"}
Nov 02 21:40:13.928016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.927987    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2" event={"ID":"48771cb7-31e2-4192-99a2-e0e36f6d8737","Type":"ContainerDied","Data":"41f3bd0ca9d6a2fb9bb98b795580059e64dfd6dc176da2563bb4998d48481248"}
Nov 02 21:40:13.928016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.927951    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"
Nov 02 21:40:13.929963 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.929940    4528 generic.go:356] "Generic (PLEG): container finished" podID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" containerID="7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca" exitCode=0
Nov 02 21:40:13.930073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.929966    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" event={"ID":"2ae9bbaf-405b-43a8-b353-73a2265a1ad3","Type":"ContainerDied","Data":"7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca"}
Nov 02 21:40:13.930073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.930005    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"
Nov 02 21:40:13.930073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.930024    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99" event={"ID":"2ae9bbaf-405b-43a8-b353-73a2265a1ad3","Type":"ContainerDied","Data":"ab5e46bca66d274cc1d10573dd7d878eb8f59927f30f62bea1c668081b9b43bd"}
Nov 02 21:40:13.942806 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.942781    4528 scope.go:122] "RemoveContainer" containerID="ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7"
Nov 02 21:40:13.943474 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:40:13.943283    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7\": container with ID starting with ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7 not found: ID does not exist" containerID="ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7"
Nov 02 21:40:13.943474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.943322    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7"} err="failed to get container status \"ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7\": rpc error: code = NotFound desc = could not find container \"ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7\": container with ID starting with ec8c0d196147f0991a8d3acb6fa1cd330d453c39271b9878a5aa55204e5475a7 not found: ID does not exist"
Nov 02 21:40:13.943474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.943346    4528 scope.go:122] "RemoveContainer" containerID="1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601"
Nov 02 21:40:13.955997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.955976    4528 scope.go:122] "RemoveContainer" containerID="1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601"
Nov 02 21:40:13.956715 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:40:13.956693    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601\": container with ID starting with 1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601 not found: ID does not exist" containerID="1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601"
Nov 02 21:40:13.956805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.956722    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601"} err="failed to get container status \"1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601\": rpc error: code = NotFound desc = could not find container \"1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601\": container with ID starting with 1504fa9eda00be0ffb96d105e742fd031670735d1ee92033c13003a2d59c9601 not found: ID does not exist"
Nov 02 21:40:13.956805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.956743    4528 scope.go:122] "RemoveContainer" containerID="7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca"
Nov 02 21:40:13.963586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.963558    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xkh\" (UniqueName: \"kubernetes.io/projected/3aac0858-7c8f-4144-b5b5-ed9389162e81-kube-api-access-75xkh\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq\" (UID: \"3aac0858-7c8f-4144-b5b5-ed9389162e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:13.963791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.963772    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aac0858-7c8f-4144-b5b5-ed9389162e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq\" (UID: \"3aac0858-7c8f-4144-b5b5-ed9389162e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:13.964693 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.964669    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aac0858-7c8f-4144-b5b5-ed9389162e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq\" (UID: \"3aac0858-7c8f-4144-b5b5-ed9389162e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:13.966690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.966674    4528 scope.go:122] "RemoveContainer" containerID="ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150"
Nov 02 21:40:13.975319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.975301    4528 scope.go:122] "RemoveContainer" containerID="7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca"
Nov 02 21:40:13.975669 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:40:13.975649    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca\": container with ID starting with 7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca not found: ID does not exist" containerID="7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca"
Nov 02 21:40:13.975748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.975680    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca"} err="failed to get container status \"7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca\": rpc error: code = NotFound desc = could not find container \"7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca\": container with ID starting with 7f1daa01cc1a2c96f9b60bf31a1ff13f4df6de6b401c5c1bef867974172370ca not found: ID does not exist"
Nov 02 21:40:13.975748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.975706    4528 scope.go:122] "RemoveContainer" containerID="ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150"
Nov 02 21:40:13.975979 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:40:13.975953    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150\": container with ID starting with ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150 not found: ID does not exist" containerID="ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150"
Nov 02 21:40:13.976094 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:13.975981    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150"} err="failed to get container status \"ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150\": rpc error: code = NotFound desc = could not find container \"ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150\": container with ID starting with ab2789a41da14933148cf9a8c7d6c88368b2707c5f127f97fec286bc8991a150 not found: ID does not exist"
Nov 02 21:40:14.020396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.020366    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xkh\" (UniqueName: \"kubernetes.io/projected/3aac0858-7c8f-4144-b5b5-ed9389162e81-kube-api-access-75xkh\") pod \"payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq\" (UID: \"3aac0858-7c8f-4144-b5b5-ed9389162e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:14.085534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.085500    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:14.113871 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:40:14.113837    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:40:14.289263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.289232    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"]
Nov 02 21:40:14.294045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.294021    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-wzf99"]
Nov 02 21:40:14.404075 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.404041    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"]
Nov 02 21:40:14.411125 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.407830    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-trzz4"]
Nov 02 21:40:14.441424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.441395    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"]
Nov 02 21:40:14.446470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.446451    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:14.460643 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:14.460615    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aac0858_7c8f_4144_b5b5_ed9389162e81.slice/crio-1a7ca55402160bbff9a812846cbcf105ec8dafaa7b8d801f34ae89bacd5b7e67 WatchSource:0}: Error finding container 1a7ca55402160bbff9a812846cbcf105ec8dafaa7b8d801f34ae89bacd5b7e67: Status 404 returned error can't find the container with id 1a7ca55402160bbff9a812846cbcf105ec8dafaa7b8d801f34ae89bacd5b7e67
Nov 02 21:40:14.462946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.462925    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"]
Nov 02 21:40:14.499607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.499582    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"]
Nov 02 21:40:14.519196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.518773    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"]
Nov 02 21:40:14.522781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.522758    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-nqph2"]
Nov 02 21:40:14.571348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.571278    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt\" (UID: \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:14.571480 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.571434    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2ks\" (UniqueName: \"kubernetes.io/projected/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-kube-api-access-vf2ks\") pod \"payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt\" (UID: \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:14.672160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.672126    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2ks\" (UniqueName: \"kubernetes.io/projected/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-kube-api-access-vf2ks\") pod \"payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt\" (UID: \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:14.672527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.672266    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt\" (UID: \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:14.672924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.672896    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt\" (UID: \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:14.730157 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.730126    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2ks\" (UniqueName: \"kubernetes.io/projected/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-kube-api-access-vf2ks\") pod \"payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt\" (UID: \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:14.762145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.762093    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:14.914627 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.914588    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae9bbaf-405b-43a8-b353-73a2265a1ad3" path="/var/lib/kubelet/pods/2ae9bbaf-405b-43a8-b353-73a2265a1ad3/volumes"
Nov 02 21:40:14.915231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.915215    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48771cb7-31e2-4192-99a2-e0e36f6d8737" path="/var/lib/kubelet/pods/48771cb7-31e2-4192-99a2-e0e36f6d8737/volumes"
Nov 02 21:40:14.915774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.915760    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe5a031-de99-4fb9-b118-68ba39cf459b" path="/var/lib/kubelet/pods/ebe5a031-de99-4fb9-b118-68ba39cf459b/volumes"
Nov 02 21:40:14.940716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.940687    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq" event={"ID":"3aac0858-7c8f-4144-b5b5-ed9389162e81","Type":"ContainerStarted","Data":"2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb"}
Nov 02 21:40:14.940841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.940746    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq" event={"ID":"3aac0858-7c8f-4144-b5b5-ed9389162e81","Type":"ContainerStarted","Data":"1a7ca55402160bbff9a812846cbcf105ec8dafaa7b8d801f34ae89bacd5b7e67"}
Nov 02 21:40:14.940930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:14.940871    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:15.058716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.058665    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq" podStartSLOduration=2.058650969 podStartE2EDuration="2.058650969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:15.055891509 +0000 UTC m=+1902.319427825" watchObservedRunningTime="2026-11-02 21:40:15.058650969 +0000 UTC m=+1902.322187286"
Nov 02 21:40:15.139729 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:15.139694    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d5b1cb_44dc_4659_8ac4_38e5d115cad0.slice/crio-5664aa07106b7ba16103aca48858f98f5cfa7760f22fb4a90ee6f1ffcfba0601 WatchSource:0}: Error finding container 5664aa07106b7ba16103aca48858f98f5cfa7760f22fb4a90ee6f1ffcfba0601: Status 404 returned error can't find the container with id 5664aa07106b7ba16103aca48858f98f5cfa7760f22fb4a90ee6f1ffcfba0601
Nov 02 21:40:15.142442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.142418    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"]
Nov 02 21:40:15.410303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.410219    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"]
Nov 02 21:40:15.414438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.414411    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.467025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.466998    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:40:15.467191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.467163    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:40:15.467255 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.467239    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:40:15.467306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.467255    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:40:15.467374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.467336    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"]
Nov 02 21:40:15.487035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.487003    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06cab2a9-tchsz\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.487176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.487062    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06cab2a9-tchsz\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.487329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.487306    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662bb\" (UniqueName: \"kubernetes.io/projected/acc30884-92a0-484e-a7a6-be87b920e81b-kube-api-access-662bb\") pod \"maas-api-revoke-sub-06cab2a9-tchsz\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.588841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.588808    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06cab2a9-tchsz\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.589055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.588888    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06cab2a9-tchsz\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.589212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.589194    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662bb\" (UniqueName: \"kubernetes.io/projected/acc30884-92a0-484e-a7a6-be87b920e81b-kube-api-access-662bb\") pod \"maas-api-revoke-sub-06cab2a9-tchsz\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.590270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.590248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06cab2a9-tchsz\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.591573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.591548    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06cab2a9-tchsz\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.646002 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.645968    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662bb\" (UniqueName: \"kubernetes.io/projected/acc30884-92a0-484e-a7a6-be87b920e81b-kube-api-access-662bb\") pod \"maas-api-revoke-sub-06cab2a9-tchsz\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.730219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.728188    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:15.953489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.953451    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt" event={"ID":"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0","Type":"ContainerStarted","Data":"8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1"}
Nov 02 21:40:15.953670 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.953514    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt" event={"ID":"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0","Type":"ContainerStarted","Data":"5664aa07106b7ba16103aca48858f98f5cfa7760f22fb4a90ee6f1ffcfba0601"}
Nov 02 21:40:15.953868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:15.953841    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:16.069930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:16.069878    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt" podStartSLOduration=2.069863114 podStartE2EDuration="2.069863114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:16.067547853 +0000 UTC m=+1903.331084170" watchObservedRunningTime="2026-11-02 21:40:16.069863114 +0000 UTC m=+1903.333399432"
Nov 02 21:40:16.118677 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:16.118645    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"]
Nov 02 21:40:16.151467 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:16.151441    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc30884_92a0_484e_a7a6_be87b920e81b.slice/crio-d788d5c9bf0a7a64682bc398ec9ecd3b941665c9a310a225819746ef63cc8150 WatchSource:0}: Error finding container d788d5c9bf0a7a64682bc398ec9ecd3b941665c9a310a225819746ef63cc8150: Status 404 returned error can't find the container with id d788d5c9bf0a7a64682bc398ec9ecd3b941665c9a310a225819746ef63cc8150
Nov 02 21:40:16.880647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:16.880622    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:40:16.881033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:16.880683    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:40:16.960899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:16.960867    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz" event={"ID":"acc30884-92a0-484e-a7a6-be87b920e81b","Type":"ContainerStarted","Data":"55817c94e6890c8c5d5f3f17ae0d14ffdbc5c0a6e7da57e0a94b8394bfec3fcf"}
Nov 02 21:40:16.961084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:16.960924    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz" event={"ID":"acc30884-92a0-484e-a7a6-be87b920e81b","Type":"ContainerStarted","Data":"d788d5c9bf0a7a64682bc398ec9ecd3b941665c9a310a225819746ef63cc8150"}
Nov 02 21:40:17.227234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:17.227134    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz" podStartSLOduration=2.227119597 podStartE2EDuration="2.227119597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:17.224274285 +0000 UTC m=+1904.487810601" watchObservedRunningTime="2026-11-02 21:40:17.227119597 +0000 UTC m=+1904.490655905"
Nov 02 21:40:17.969379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:17.969343    4528 generic.go:356] "Generic (PLEG): container finished" podID="acc30884-92a0-484e-a7a6-be87b920e81b" containerID="55817c94e6890c8c5d5f3f17ae0d14ffdbc5c0a6e7da57e0a94b8394bfec3fcf" exitCode=0
Nov 02 21:40:17.969758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:17.969403    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz" event={"ID":"acc30884-92a0-484e-a7a6-be87b920e81b","Type":"ContainerDied","Data":"55817c94e6890c8c5d5f3f17ae0d14ffdbc5c0a6e7da57e0a94b8394bfec3fcf"}
Nov 02 21:40:18.914951 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:18.914921    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:40:19.127843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.127819    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:19.250278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.250196    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-xks-ca\") pod \"acc30884-92a0-484e-a7a6-be87b920e81b\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") "
Nov 02 21:40:19.250434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.250364    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-openshift-ca\") pod \"acc30884-92a0-484e-a7a6-be87b920e81b\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") "
Nov 02 21:40:19.250500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.250474    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc30884-92a0-484e-a7a6-be87b920e81b-kube-api-access-662bb\" (UniqueName: \"kubernetes.io/projected/acc30884-92a0-484e-a7a6-be87b920e81b-kube-api-access-662bb\") pod \"acc30884-92a0-484e-a7a6-be87b920e81b\" (UID: \"acc30884-92a0-484e-a7a6-be87b920e81b\") "
Nov 02 21:40:19.250824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.250790    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-openshift-ca" pod "acc30884-92a0-484e-a7a6-be87b920e81b" (UID: "acc30884-92a0-484e-a7a6-be87b920e81b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:40:19.250985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.250962    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:19.252799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.252767    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-xks-ca" pod "acc30884-92a0-484e-a7a6-be87b920e81b" (UID: "acc30884-92a0-484e-a7a6-be87b920e81b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:40:19.253289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.253243    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc30884-92a0-484e-a7a6-be87b920e81b-kube-api-access-662bb" pod "acc30884-92a0-484e-a7a6-be87b920e81b" (UID: "acc30884-92a0-484e-a7a6-be87b920e81b"). InnerVolumeSpecName "kube-api-access-662bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:19.352439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.352406    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-662bb\" (UniqueName: \"kubernetes.io/projected/acc30884-92a0-484e-a7a6-be87b920e81b-kube-api-access-662bb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:19.352439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.352436    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc30884-92a0-484e-a7a6-be87b920e81b-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:19.985899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.985849    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz" event={"ID":"acc30884-92a0-484e-a7a6-be87b920e81b","Type":"ContainerDied","Data":"d788d5c9bf0a7a64682bc398ec9ecd3b941665c9a310a225819746ef63cc8150"}
Nov 02 21:40:19.986090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.985920    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d788d5c9bf0a7a64682bc398ec9ecd3b941665c9a310a225819746ef63cc8150"
Nov 02 21:40:19.986090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:19.986011    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"
Nov 02 21:40:25.955319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:25.955290    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:40:26.963303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:26.963265    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:40:34.257598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:34.257571    4528 scope.go:122] "RemoveContainer" containerID="6a1f11c32e5514d55cfe4acea051d2008d308353e4247972b583e86a100a660e"
Nov 02 21:40:38.048072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.048038    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"]
Nov 02 21:40:38.048871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.048851    4528 cpu_manager.go:414] "containerMap: removing container" podUID="acc30884-92a0-484e-a7a6-be87b920e81b" containerName="revoke-keys"
Nov 02 21:40:38.049000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.048873    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc30884-92a0-484e-a7a6-be87b920e81b" containerName="revoke-keys"
Nov 02 21:40:38.049057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.049046    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc30884-92a0-484e-a7a6-be87b920e81b" containerName="revoke-keys"
Nov 02 21:40:38.054627 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.054609    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.119186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.117699    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3e3997-openshift-default-dockercfg-fstqp\""
Nov 02 21:40:38.141137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.139088    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-credential-socket\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.141137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.139215    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-socket\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.141137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.139260    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istiod-ca-cert\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.141137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.139327    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-token\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.141137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.139353    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8khc\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-kube-api-access-v8khc\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.141137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.139396    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-certs\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.141137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.139428    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-podinfo\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.141137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.139465    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-envoy\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.141137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.139506    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-data\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.255456    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-credential-socket\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.255551    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-socket\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.255591    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istiod-ca-cert\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.255649    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-token\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.255766    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8khc\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-kube-api-access-v8khc\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.255810    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-certs\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.255842    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-podinfo\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.255905    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-envoy\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.255969    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-data\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.256492    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-data\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.256823    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-credential-socket\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.256996    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-certs\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.265130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.257074    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-socket\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.296891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.281904    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istiod-ca-cert\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.302834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.302768    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-podinfo\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.309049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.307208    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"]
Nov 02 21:40:38.324727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.324699    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8khc\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-kube-api-access-v8khc\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.325319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.325276    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-token\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.326924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.326883    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-envoy\") pod \"e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") " pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.368188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.368162    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:38.828273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.827372    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"]
Nov 02 21:40:38.835501 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:38.829813    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1acbfd2_7cae_40a9_bf4f_ac7f3f8c99ef.slice/crio-e20f10f4e87847d174b8bcf7dedd9422d0a4a2a7ea905da4eeb37825589956b5 WatchSource:0}: Error finding container e20f10f4e87847d174b8bcf7dedd9422d0a4a2a7ea905da4eeb37825589956b5: Status 404 returned error can't find the container with id e20f10f4e87847d174b8bcf7dedd9422d0a4a2a7ea905da4eeb37825589956b5
Nov 02 21:40:38.835607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.833368    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:38.835784 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.835759    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:38.835946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.835921    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:38.902389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.902365    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"]
Nov 02 21:40:38.907736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.907712    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:38.962173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.960123    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"]
Nov 02 21:40:38.973492 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.973461    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ce2453\""
Nov 02 21:40:38.973890 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:38.973868    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ce2453-dockercfg-fs59l\""
Nov 02 21:40:39.093212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.091613    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be25b2b4-327d-46f0-b23e-72b71ed06af0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f\" (UID: \"be25b2b4-327d-46f0-b23e-72b71ed06af0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:39.093212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.091665    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9h6q\" (UniqueName: \"kubernetes.io/projected/be25b2b4-327d-46f0-b23e-72b71ed06af0-kube-api-access-c9h6q\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f\" (UID: \"be25b2b4-327d-46f0-b23e-72b71ed06af0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:39.192585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.192555    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be25b2b4-327d-46f0-b23e-72b71ed06af0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f\" (UID: \"be25b2b4-327d-46f0-b23e-72b71ed06af0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:39.192756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.192629    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9h6q\" (UniqueName: \"kubernetes.io/projected/be25b2b4-327d-46f0-b23e-72b71ed06af0-kube-api-access-c9h6q\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f\" (UID: \"be25b2b4-327d-46f0-b23e-72b71ed06af0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:39.193249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.193229    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be25b2b4-327d-46f0-b23e-72b71ed06af0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f\" (UID: \"be25b2b4-327d-46f0-b23e-72b71ed06af0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:39.249601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.249568    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9h6q\" (UniqueName: \"kubernetes.io/projected/be25b2b4-327d-46f0-b23e-72b71ed06af0-kube-api-access-c9h6q\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f\" (UID: \"be25b2b4-327d-46f0-b23e-72b71ed06af0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:39.277825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.277792    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" event={"ID":"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef","Type":"ContainerStarted","Data":"2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06"}
Nov 02 21:40:39.278024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.277843    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" event={"ID":"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef","Type":"ContainerStarted","Data":"e20f10f4e87847d174b8bcf7dedd9422d0a4a2a7ea905da4eeb37825589956b5"}
Nov 02 21:40:39.369254 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.368888    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:39.371810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.371775    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 21:40:39.371955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.371833    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 21:40:39.396382 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.396335    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podStartSLOduration=1.396321592 podStartE2EDuration="1.396321592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:39.393842266 +0000 UTC m=+1926.657378585" watchObservedRunningTime="2026-11-02 21:40:39.396321592 +0000 UTC m=+1926.659857909"
Nov 02 21:40:39.540632 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.539381    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:39.931469 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:39.931438    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe25b2b4_327d_46f0_b23e_72b71ed06af0.slice/crio-04a9d32953a3c265799b8560358b337154431783297b82638f0ea15252c868a6 WatchSource:0}: Error finding container 04a9d32953a3c265799b8560358b337154431783297b82638f0ea15252c868a6: Status 404 returned error can't find the container with id 04a9d32953a3c265799b8560358b337154431783297b82638f0ea15252c868a6
Nov 02 21:40:39.933620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:39.933595    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"]
Nov 02 21:40:40.285168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:40.285135    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f" event={"ID":"be25b2b4-327d-46f0-b23e-72b71ed06af0","Type":"ContainerStarted","Data":"ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0"}
Nov 02 21:40:40.285655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:40.285192    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f" event={"ID":"be25b2b4-327d-46f0-b23e-72b71ed06af0","Type":"ContainerStarted","Data":"04a9d32953a3c265799b8560358b337154431783297b82638f0ea15252c868a6"}
Nov 02 21:40:40.285655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:40.285327    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:40.373690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:40.373659    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:40.420445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:40.419372    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f" podStartSLOduration=2.419354339 podStartE2EDuration="2.419354339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:40.419014074 +0000 UTC m=+1927.682550394" watchObservedRunningTime="2026-11-02 21:40:40.419354339 +0000 UTC m=+1927.682890667"
Nov 02 21:40:41.291658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:41.291628    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:41.292423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:41.292406    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:40:43.568538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.568425    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"]
Nov 02 21:40:43.581659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.581630    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:43.621959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.621924    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"]
Nov 02 21:40:43.669074 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.669040    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-387f88\""
Nov 02 21:40:43.669242 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.669200    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-387f88-dockercfg-zflqd\""
Nov 02 21:40:43.766720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.766675    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxhgr\" (UniqueName: \"kubernetes.io/projected/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-kube-api-access-sxhgr\") pod \"payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv\" (UID: \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:43.766720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.766726    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv\" (UID: \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:43.868481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.868398    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhgr\" (UniqueName: \"kubernetes.io/projected/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-kube-api-access-sxhgr\") pod \"payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv\" (UID: \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:43.868481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.868437    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv\" (UID: \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:43.869162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.869137    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv\" (UID: \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:43.925672 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:43.925639    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhgr\" (UniqueName: \"kubernetes.io/projected/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-kube-api-access-sxhgr\") pod \"payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv\" (UID: \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:44.112718 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:40:44.112684    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:40:44.201387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:44.201306    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:44.582973 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:44.582948    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"]
Nov 02 21:40:45.321482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:45.321439    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv" event={"ID":"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7","Type":"ContainerStarted","Data":"c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75"}
Nov 02 21:40:45.321644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:45.321506    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv" event={"ID":"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7","Type":"ContainerStarted","Data":"d50aad885b7d3cf8b84b76da12d85915839c1ae0541bdfac36d7466e44a2a800"}
Nov 02 21:40:45.321644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:45.321537    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:45.436457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:45.436406    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv" podStartSLOduration=2.43638879 podStartE2EDuration="2.43638879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:45.436084291 +0000 UTC m=+1932.699620607" watchObservedRunningTime="2026-11-02 21:40:45.43638879 +0000 UTC m=+1932.699925106"
Nov 02 21:40:46.206155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:46.206122    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"]
Nov 02 21:40:46.304092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:46.304062    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"]
Nov 02 21:40:47.905240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:47.905201    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:40:49.533353 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.533309    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"]
Nov 02 21:40:49.539168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.539141    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.559977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.559952    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.560095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.560001    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.560095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.560068    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.560208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.560178    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.560208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.560200    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tf4b\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-kube-api-access-7tf4b\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.560298 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.560239    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.560371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.560349    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.560429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.560395    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.560485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.560455    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.584318 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.584291    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"]
Nov 02 21:40:49.589301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.589285    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"]
Nov 02 21:40:49.589423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.589409    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.593272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.593253    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.631245 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.631215    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4lw9\""
Nov 02 21:40:49.637098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.637067    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"]
Nov 02 21:40:49.662249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662216    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.662409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662264    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.662568 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662547    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.662664 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662632    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.662664 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662647    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.662778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662711    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.662778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662740    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573adb50-82ab-442f-a850-ce34862f2b10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.662778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662768    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.662943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662792    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.662943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662840    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.662943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662863    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.662943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662884    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.662943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662900    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.662943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662941    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662971    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573adb50-82ab-442f-a850-ce34862f2b10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.662985    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j4sn\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-kube-api-access-7j4sn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663006    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663024    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tf4b\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-kube-api-access-7tf4b\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663067    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663081    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr746\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-kube-api-access-jr746\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663133    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663160    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663187    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.663270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663247    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.663659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663281    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.663659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663317    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.663659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663360    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.663659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.663415    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.664151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.664127    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.664382 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.664366    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.664993 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.664968    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.665340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.665316    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.666969 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.666951    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.667452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.667429    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.691816 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.691779    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"]
Nov 02 21:40:49.698671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.698643    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"]
Nov 02 21:40:49.730343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.730306    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tf4b\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-kube-api-access-7tf4b\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.732359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.732333    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-chvg9\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.764525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764497    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.764638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764536    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.764638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764559    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.764638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764581    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.764638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764609    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573adb50-82ab-442f-a850-ce34862f2b10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.764638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764627    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.764892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764652    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.764892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764668    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.764892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764689    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764976    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.764986    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.765078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765005    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.765314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765138    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.765314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765180    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573adb50-82ab-442f-a850-ce34862f2b10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765254    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.765489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765331    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.765489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765355    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.765489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765365    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j4sn\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-kube-api-access-7j4sn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765386    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.765489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765459    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765487    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr746\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-kube-api-access-jr746\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.765760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765583    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.765760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765620    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765661    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765779    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573adb50-82ab-442f-a850-ce34862f2b10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765781    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.765994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.765978    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.767458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.767422    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573adb50-82ab-442f-a850-ce34862f2b10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.767757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.767734    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.767833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.767816    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.768035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.768017    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.830842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.830755    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr746\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-kube-api-access-jr746\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.830842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.830791    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b65pw\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:49.831205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.831183    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j4sn\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-kube-api-access-7j4sn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.831614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.831594    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-n8hjs\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.853289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.853262    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:49.932310 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.932253    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:49.946790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:49.942398    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:50.341753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.341719    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"]
Nov 02 21:40:50.375061 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:50.375036    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683ffcec_8a51_4714_9a3e_a0f7c58a0ce7.slice/crio-abdefbf2c1e481e9c4ff8bf8a0cead89ff708b6bcff5429db7c17633d6401b68 WatchSource:0}: Error finding container abdefbf2c1e481e9c4ff8bf8a0cead89ff708b6bcff5429db7c17633d6401b68: Status 404 returned error can't find the container with id abdefbf2c1e481e9c4ff8bf8a0cead89ff708b6bcff5429db7c17633d6401b68
Nov 02 21:40:50.377735 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.377699    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:50.377841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.377767    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:50.377841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.377796    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:50.389242 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:50.389216    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bc3d67_6e03_4bbb_8316_4f28a4a6b111.slice/crio-7d5925220c40f1f5da5d2f16239d514dd6beae4334ef605c3d507af4fd2644f3 WatchSource:0}: Error finding container 7d5925220c40f1f5da5d2f16239d514dd6beae4334ef605c3d507af4fd2644f3: Status 404 returned error can't find the container with id 7d5925220c40f1f5da5d2f16239d514dd6beae4334ef605c3d507af4fd2644f3
Nov 02 21:40:50.390766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.390702    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"]
Nov 02 21:40:50.391790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.391761    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:50.391891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.391832    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:50.391891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.391877    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:50.438133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.438093    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"]
Nov 02 21:40:50.440434 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:40:50.440407    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573adb50_82ab_442f_a850_ce34862f2b10.slice/crio-68f464b15c4bd9b32683c91b21931d3a6875e790b5274f0a2a82b3da677d25aa WatchSource:0}: Error finding container 68f464b15c4bd9b32683c91b21931d3a6875e790b5274f0a2a82b3da677d25aa: Status 404 returned error can't find the container with id 68f464b15c4bd9b32683c91b21931d3a6875e790b5274f0a2a82b3da677d25aa
Nov 02 21:40:50.443962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.443565    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:50.443962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.443631    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:50.443962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:50.443658    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:40:51.207137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.206624    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt" podUID="91b46a41-5785-4ed1-ba52-eae364782587" containerName="payload-processing" containerID="cri-o://cf936d0bebbda525f117f16a7ff30b74a174ce459879a9e137717985382d1bc0" gracePeriod=25
Nov 02 21:40:51.321140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.311581    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g" podUID="4951470f-4373-4bce-a1db-f95395dbc27e" containerName="payload-pre-processing" containerID="cri-o://c60f212b01922256651e27b46672b0383d932d723e8b10af02ecfb424889f6fa" gracePeriod=25
Nov 02 21:40:51.321140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.317351    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:40:51.384588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.384409    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" event={"ID":"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7","Type":"ContainerStarted","Data":"56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e"}
Nov 02 21:40:51.384588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.384473    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" event={"ID":"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7","Type":"ContainerStarted","Data":"abdefbf2c1e481e9c4ff8bf8a0cead89ff708b6bcff5429db7c17633d6401b68"}
Nov 02 21:40:51.389567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.389536    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" event={"ID":"573adb50-82ab-442f-a850-ce34862f2b10","Type":"ContainerStarted","Data":"e6f4fefafe3102b7ae9fcbe2024f778377e1b6f17fc9a57b693fa0247ea36ab8"}
Nov 02 21:40:51.389754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.389737    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" event={"ID":"573adb50-82ab-442f-a850-ce34862f2b10","Type":"ContainerStarted","Data":"68f464b15c4bd9b32683c91b21931d3a6875e790b5274f0a2a82b3da677d25aa"}
Nov 02 21:40:51.392493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.392448    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" event={"ID":"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111","Type":"ContainerStarted","Data":"8709fbd4be1ba741a9300799e04cbd791397f9718ba432c386de471832e68365"}
Nov 02 21:40:51.392669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.392651    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" event={"ID":"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111","Type":"ContainerStarted","Data":"7d5925220c40f1f5da5d2f16239d514dd6beae4334ef605c3d507af4fd2644f3"}
Nov 02 21:40:51.690135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.689153    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podStartSLOduration=2.68913088 podStartE2EDuration="2.68913088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:51.5835085 +0000 UTC m=+1938.847044817" watchObservedRunningTime="2026-11-02 21:40:51.68913088 +0000 UTC m=+1938.952667200"
Nov 02 21:40:51.748481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.745609    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podStartSLOduration=2.745582823 podStartE2EDuration="2.745582823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:51.726912251 +0000 UTC m=+1938.990448568" watchObservedRunningTime="2026-11-02 21:40:51.745582823 +0000 UTC m=+1939.009119141"
Nov 02 21:40:51.891089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.883185    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:40:51.908363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.908270    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-chvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 21:40:51.908363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.908336    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 21:40:52.003333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:51.978717    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:40:52.003333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.002228    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:40:52.030133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.019996    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-n8hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 21:40:52.030133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.020056    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 21:40:52.030133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.020166    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b65pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 21:40:52.030133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.020185    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 21:40:52.235190 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.229219    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:52.282578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.273191    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:52.389132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.384224    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:52.547144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.531261    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded"
Nov 02 21:40:52.576128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.575767    4528 generic.go:356] "Generic (PLEG): container finished" podID="91b46a41-5785-4ed1-ba52-eae364782587" containerID="cf936d0bebbda525f117f16a7ff30b74a174ce459879a9e137717985382d1bc0" exitCode=0
Nov 02 21:40:52.576128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.575910    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt" event={"ID":"91b46a41-5785-4ed1-ba52-eae364782587","Type":"ContainerDied","Data":"cf936d0bebbda525f117f16a7ff30b74a174ce459879a9e137717985382d1bc0"}
Nov 02 21:40:52.716231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.716187    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:52.796449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.776671    4528 generic.go:356] "Generic (PLEG): container finished" podID="4951470f-4373-4bce-a1db-f95395dbc27e" containerID="c60f212b01922256651e27b46672b0383d932d723e8b10af02ecfb424889f6fa" exitCode=0
Nov 02 21:40:52.796449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.781887    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g" event={"ID":"4951470f-4373-4bce-a1db-f95395dbc27e","Type":"ContainerDied","Data":"c60f212b01922256651e27b46672b0383d932d723e8b10af02ecfb424889f6fa"}
Nov 02 21:40:52.816184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.797943    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:40:52.823129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.820746    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b46a41-5785-4ed1-ba52-eae364782587-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b46a41-5785-4ed1-ba52-eae364782587-plugins-config-volume\") pod \"91b46a41-5785-4ed1-ba52-eae364782587\" (UID: \"91b46a41-5785-4ed1-ba52-eae364782587\") "
Nov 02 21:40:52.823129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.820851    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b46a41-5785-4ed1-ba52-eae364782587-kube-api-access-gfn8f\" (UniqueName: \"kubernetes.io/projected/91b46a41-5785-4ed1-ba52-eae364782587-kube-api-access-gfn8f\") pod \"91b46a41-5785-4ed1-ba52-eae364782587\" (UID: \"91b46a41-5785-4ed1-ba52-eae364782587\") "
Nov 02 21:40:52.842701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.842415    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b46a41-5785-4ed1-ba52-eae364782587-plugins-config-volume" pod "91b46a41-5785-4ed1-ba52-eae364782587" (UID: "91b46a41-5785-4ed1-ba52-eae364782587"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:40:52.872646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.867969    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-chvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 21:40:52.872646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.868026    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 21:40:52.930500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.917156    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b46a41-5785-4ed1-ba52-eae364782587-kube-api-access-gfn8f" pod "91b46a41-5785-4ed1-ba52-eae364782587" (UID: "91b46a41-5785-4ed1-ba52-eae364782587"). InnerVolumeSpecName "kube-api-access-gfn8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:52.930500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.926061    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b46a41-5785-4ed1-ba52-eae364782587-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:52.930500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.926094    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfn8f\" (UniqueName: \"kubernetes.io/projected/91b46a41-5785-4ed1-ba52-eae364782587-kube-api-access-gfn8f\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:52.975031 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.974997    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:40:52.979244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.978479    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b65pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 21:40:52.983130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.979460    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 21:40:53.007180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.979936    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-n8hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 21:40:53.007443 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:52.998743    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podStartSLOduration=3.9985512659999998 podStartE2EDuration="3.998551266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:51.864398283 +0000 UTC m=+1939.127934601" watchObservedRunningTime="2026-11-02 21:40:52.998551266 +0000 UTC m=+1940.262087583"
Nov 02 21:40:53.007997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.007962    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 21:40:53.030139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.029167    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4951470f-4373-4bce-a1db-f95395dbc27e-kube-api-access-8l9tw\" (UniqueName: \"kubernetes.io/projected/4951470f-4373-4bce-a1db-f95395dbc27e-kube-api-access-8l9tw\") pod \"4951470f-4373-4bce-a1db-f95395dbc27e\" (UID: \"4951470f-4373-4bce-a1db-f95395dbc27e\") "
Nov 02 21:40:53.030139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.029397    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4951470f-4373-4bce-a1db-f95395dbc27e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4951470f-4373-4bce-a1db-f95395dbc27e-plugins-config-volume\") pod \"4951470f-4373-4bce-a1db-f95395dbc27e\" (UID: \"4951470f-4373-4bce-a1db-f95395dbc27e\") "
Nov 02 21:40:53.055957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.055911    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4951470f-4373-4bce-a1db-f95395dbc27e-plugins-config-volume" pod "4951470f-4373-4bce-a1db-f95395dbc27e" (UID: "4951470f-4373-4bce-a1db-f95395dbc27e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:40:53.103495 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.103401    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4951470f-4373-4bce-a1db-f95395dbc27e-kube-api-access-8l9tw" pod "4951470f-4373-4bce-a1db-f95395dbc27e" (UID: "4951470f-4373-4bce-a1db-f95395dbc27e"). InnerVolumeSpecName "kube-api-access-8l9tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:40:53.137135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.131129    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l9tw\" (UniqueName: \"kubernetes.io/projected/4951470f-4373-4bce-a1db-f95395dbc27e-kube-api-access-8l9tw\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:53.138372 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.138347    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4951470f-4373-4bce-a1db-f95395dbc27e-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:40:53.153264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.153191    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:53.301386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.255475    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 02 21:40:53.301386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.255901    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:40:53.301386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.298786    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:53.301386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.299141    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:40:53.405195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.395995    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 21:40:53.405195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.397149    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:53.405195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.397504    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:40:53.508609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.508217    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:53.717415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.717307    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:53.728182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.718344    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:40:53.938135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.929790    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-chvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 21:40:53.938135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.929852    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 21:40:53.964135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.960281    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-n8hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 21:40:53.968179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.964382    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 21:40:53.972191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.963206    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b65pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 21:40:53.992132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:53.972359    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 21:40:54.059858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.029124    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 21:40:54.059858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.030294    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Nov 02 21:40:54.519868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.519802    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:54.526133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.521369    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:40:54.621748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.590778    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt" event={"ID":"91b46a41-5785-4ed1-ba52-eae364782587","Type":"ContainerDied","Data":"2409dccc6096d373526d1e573d4f547baed836a09b976097b8949f47d9d61a31"}
Nov 02 21:40:54.621748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.529543    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"
Nov 02 21:40:54.777198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.773173    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 02 21:40:54.821230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.690542    4528 scope.go:122] "RemoveContainer" containerID="cf936d0bebbda525f117f16a7ff30b74a174ce459879a9e137717985382d1bc0"
Nov 02 21:40:54.837680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.837638    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:54.907130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.883874    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"
Nov 02 21:40:55.122901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.894987    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-chvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 21:40:55.122901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.103480    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 21:40:55.122901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.044316    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:55.122901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:54.903836    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g" event={"ID":"4951470f-4373-4bce-a1db-f95395dbc27e","Type":"ContainerDied","Data":"32d0811746aa056513df35e8bc08350cc7dcc004e8081a0080e8319192373834"}
Nov 02 21:40:55.122901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.023223    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b65pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 21:40:55.122901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.108780    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 21:40:55.122901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.023398    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-n8hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 21:40:55.122901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.108839    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 21:40:55.180136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.178257    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:55.430265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.430220    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:55.507014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.455801    4528 scope.go:122] "RemoveContainer" containerID="c60f212b01922256651e27b46672b0383d932d723e8b10af02ecfb424889f6fa"
Nov 02 21:40:55.507014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.491531    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:55.523288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.523250    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Nov 02 21:40:55.786669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.786644    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 21:40:55.865257 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.865215    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:55.868360 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.868331    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-chvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 21:40:55.922988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.922946    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 21:40:55.923271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.877134    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"]
Nov 02 21:40:55.982504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.954969    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b65pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 21:40:55.982858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.982819    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 21:40:55.983021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.970396    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-n8hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 21:40:55.983157 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.983132    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 21:40:55.993404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:55.993371    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 21:40:56.002193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.002152    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-7c486c96bf-pdcgt"]
Nov 02 21:40:56.090629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.090552    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"]
Nov 02 21:40:56.094868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.093821    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 21:40:56.097588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.096229    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7967b4c959-d6l4g"]
Nov 02 21:40:56.387190 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.386371    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:40:56.869488 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.869452    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-chvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 21:40:56.870271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.870236    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 21:40:56.935662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.935627    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-n8hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 21:40:56.937639 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.937589    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 21:40:56.956719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.956681    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4951470f-4373-4bce-a1db-f95395dbc27e" path="/var/lib/kubelet/pods/4951470f-4373-4bce-a1db-f95395dbc27e/volumes"
Nov 02 21:40:56.957657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.957632    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b46a41-5785-4ed1-ba52-eae364782587" path="/var/lib/kubelet/pods/91b46a41-5785-4ed1-ba52-eae364782587/volumes"
Nov 02 21:40:56.958775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.958754    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b65pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 21:40:56.958953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:56.958929    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 21:40:57.853727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:57.853672    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-chvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 21:40:57.854540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:57.854090    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 21:40:57.933540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:57.933500    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-n8hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 21:40:57.934012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:57.933562    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 21:40:57.943943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:57.943911    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b65pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 21:40:57.944088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:57.943963    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 21:40:58.204211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.204097    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"]
Nov 02 21:40:58.204423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.204392    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy" containerID="cri-o://a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529" gracePeriod=30
Nov 02 21:40:58.611365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.611258    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:40:58.611365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.611328    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:40:58.854380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.853947    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-chvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 21:40:58.854380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.854008    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 21:40:58.933268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.933177    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-n8hjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 21:40:58.933268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.933254    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 21:40:58.943544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.943511    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b65pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 21:40:58.944007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:58.943566    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 21:40:59.772485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:59.772104    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:40:59.772485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:40:59.772184    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:41:00.732137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:00.731885    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:41:00.734127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:00.732719    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:41:00.734127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:00.733916    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 21:41:00.779680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:00.779647    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:41:00.780908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:00.780882    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:41:00.781696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:00.781676    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:41:00.794046 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:00.794020    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:41:00.794484 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:00.794465    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:41:00.795428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:00.795410    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:41:01.076459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:01.076420    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"]
Nov 02 21:41:01.076761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:01.076704    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy" containerID="cri-o://9756ca8039029b7db63320b9e768072739d204172e416a3aa0590b58accfa373" gracePeriod=30
Nov 02 21:41:02.215811 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:02.215780    4528 generic.go:356] "Generic (PLEG): container finished" podID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerID="f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8" exitCode=0
Nov 02 21:41:02.216345 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:02.215852    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" event={"ID":"a7af38e3-4322-46b6-9eb6-638c9a3e5be0","Type":"ContainerDied","Data":"f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8"}
Nov 02 21:41:02.216545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:02.216528    4528 scope.go:122] "RemoveContainer" containerID="f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8"
Nov 02 21:41:02.218067 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:02.218039    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:41:02.218180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:02.218102    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:41:02.218180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:02.218151    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:41:03.240671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.240637    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" event={"ID":"a7af38e3-4322-46b6-9eb6-638c9a3e5be0","Type":"ContainerStarted","Data":"1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b"}
Nov 02 21:41:03.743029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.743004    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:41:03.743288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.743263    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 21:41:03.743387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.743307    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 21:41:03.938902 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.938873    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:41:03.967836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.967798    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-certs\") pod \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") "
Nov 02 21:41:03.967968 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.967860    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-kube-api-access-cvtr8\") pod \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") "
Nov 02 21:41:03.967968 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.967893    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-socket\") pod \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") "
Nov 02 21:41:03.967968 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.967915    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-envoy\") pod \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") "
Nov 02 21:41:03.968171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.967984    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-token\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-token\") pod \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") "
Nov 02 21:41:03.968171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.968036    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-certs" pod "a1e058f5-4c9d-41e7-a301-22e81d9270b2" (UID: "a1e058f5-4c9d-41e7-a301-22e81d9270b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:03.968171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.968049    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istiod-ca-cert\") pod \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") "
Nov 02 21:41:03.968171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.968162    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-credential-socket\") pod \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") "
Nov 02 21:41:03.968341 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.968212    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-socket" pod "a1e058f5-4c9d-41e7-a301-22e81d9270b2" (UID: "a1e058f5-4c9d-41e7-a301-22e81d9270b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:03.968341 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.968229    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-data\") pod \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") "
Nov 02 21:41:03.968437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.968340    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-podinfo\") pod \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\" (UID: \"a1e058f5-4c9d-41e7-a301-22e81d9270b2\") "
Nov 02 21:41:03.968437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.968425    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istiod-ca-cert" pod "a1e058f5-4c9d-41e7-a301-22e81d9270b2" (UID: "a1e058f5-4c9d-41e7-a301-22e81d9270b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:41:03.968554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.968532    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-data" pod "a1e058f5-4c9d-41e7-a301-22e81d9270b2" (UID: "a1e058f5-4c9d-41e7-a301-22e81d9270b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:03.968614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.968589    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-credential-socket" pod "a1e058f5-4c9d-41e7-a301-22e81d9270b2" (UID: "a1e058f5-4c9d-41e7-a301-22e81d9270b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:03.969078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.969054    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:03.969191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.969083    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:03.969191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.969101    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:03.969191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.969146    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:03.969191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.969161    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:03.970323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.970295    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-envoy" pod "a1e058f5-4c9d-41e7-a301-22e81d9270b2" (UID: "a1e058f5-4c9d-41e7-a301-22e81d9270b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:03.970323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.970317    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-token" pod "a1e058f5-4c9d-41e7-a301-22e81d9270b2" (UID: "a1e058f5-4c9d-41e7-a301-22e81d9270b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:41:03.970463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.970392    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-kube-api-access-cvtr8" pod "a1e058f5-4c9d-41e7-a301-22e81d9270b2" (UID: "a1e058f5-4c9d-41e7-a301-22e81d9270b2"). InnerVolumeSpecName "kube-api-access-cvtr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:41:03.970463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:03.970430    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-podinfo" pod "a1e058f5-4c9d-41e7-a301-22e81d9270b2" (UID: "a1e058f5-4c9d-41e7-a301-22e81d9270b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:41:04.048932 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.048895    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:41:04.049137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.048979    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:41:04.070122 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.070070    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:04.070122 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.070102    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:04.070122 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.070126    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/a1e058f5-4c9d-41e7-a301-22e81d9270b2-kube-api-access-cvtr8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:04.070419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.070137    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e058f5-4c9d-41e7-a301-22e81d9270b2-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:04.248724 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.248692    4528 generic.go:356] "Generic (PLEG): container finished" podID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerID="a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529" exitCode=0
Nov 02 21:41:04.249248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.248795    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"
Nov 02 21:41:04.249248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.248831    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" event={"ID":"a1e058f5-4c9d-41e7-a301-22e81d9270b2","Type":"ContainerDied","Data":"a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529"}
Nov 02 21:41:04.249248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.248898    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622" event={"ID":"a1e058f5-4c9d-41e7-a301-22e81d9270b2","Type":"ContainerDied","Data":"7c4344ce5b953f537441849461b304b33196376550cbd8921112e8f1fbba2b8d"}
Nov 02 21:41:04.249248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.248940    4528 scope.go:122] "RemoveContainer" containerID="a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529"
Nov 02 21:41:04.260058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.260041    4528 scope.go:122] "RemoveContainer" containerID="a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529"
Nov 02 21:41:04.260357 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:41:04.260337    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529\": container with ID starting with a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529 not found: ID does not exist" containerID="a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529"
Nov 02 21:41:04.260416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.260365    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529"} err="failed to get container status \"a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529\": rpc error: code = NotFound desc = could not find container \"a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529\": container with ID starting with a0af5901793c02a8b3d8d5c479ae0d6d0dde111c82d3db2cb4c9d06ccb010529 not found: ID does not exist"
Nov 02 21:41:04.424188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.423854    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"]
Nov 02 21:41:04.428407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.428378    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-dg622"]
Nov 02 21:41:04.744234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.744154    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 21:41:04.744234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.744204    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 21:41:04.932383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:04.932350    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" path="/var/lib/kubelet/pods/a1e058f5-4c9d-41e7-a301-22e81d9270b2/volumes"
Nov 02 21:41:05.743227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:05.743182    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 21:41:05.743621 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:05.743256    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 21:41:06.266395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.266361    4528 generic.go:356] "Generic (PLEG): container finished" podID="772fd287-d421-4a6e-86cb-c9ef846603df" containerID="9756ca8039029b7db63320b9e768072739d204172e416a3aa0590b58accfa373" exitCode=0
Nov 02 21:41:06.266555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.266436    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" event={"ID":"772fd287-d421-4a6e-86cb-c9ef846603df","Type":"ContainerDied","Data":"9756ca8039029b7db63320b9e768072739d204172e416a3aa0590b58accfa373"}
Nov 02 21:41:06.389963 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.389937    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:41:06.532717 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.532683    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-credential-socket\") pod \"772fd287-d421-4a6e-86cb-c9ef846603df\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") "
Nov 02 21:41:06.532935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.532769    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-istio-token\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-istio-token\") pod \"772fd287-d421-4a6e-86cb-c9ef846603df\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") "
Nov 02 21:41:06.532935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.532797    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-certs\") pod \"772fd287-d421-4a6e-86cb-c9ef846603df\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") "
Nov 02 21:41:06.532935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.532837    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-envoy\") pod \"772fd287-d421-4a6e-86cb-c9ef846603df\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") "
Nov 02 21:41:06.532935 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.532898    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-kube-api-access-x7pk4\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-kube-api-access-x7pk4\") pod \"772fd287-d421-4a6e-86cb-c9ef846603df\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") "
Nov 02 21:41:06.533218 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.532933    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-data\") pod \"772fd287-d421-4a6e-86cb-c9ef846603df\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") "
Nov 02 21:41:06.533218 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.532954    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-credential-socket" pod "772fd287-d421-4a6e-86cb-c9ef846603df" (UID: "772fd287-d421-4a6e-86cb-c9ef846603df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:06.533218 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.532967    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772fd287-d421-4a6e-86cb-c9ef846603df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772fd287-d421-4a6e-86cb-c9ef846603df-istiod-ca-cert\") pod \"772fd287-d421-4a6e-86cb-c9ef846603df\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") "
Nov 02 21:41:06.533390 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.533216    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-socket\") pod \"772fd287-d421-4a6e-86cb-c9ef846603df\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") "
Nov 02 21:41:06.533390 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.533256    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/772fd287-d421-4a6e-86cb-c9ef846603df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772fd287-d421-4a6e-86cb-c9ef846603df-istio-podinfo\") pod \"772fd287-d421-4a6e-86cb-c9ef846603df\" (UID: \"772fd287-d421-4a6e-86cb-c9ef846603df\") "
Nov 02 21:41:06.533390 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.533309    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772fd287-d421-4a6e-86cb-c9ef846603df-istiod-ca-cert" pod "772fd287-d421-4a6e-86cb-c9ef846603df" (UID: "772fd287-d421-4a6e-86cb-c9ef846603df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:41:06.533773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.533742    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-data" pod "772fd287-d421-4a6e-86cb-c9ef846603df" (UID: "772fd287-d421-4a6e-86cb-c9ef846603df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:06.533773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.533769    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:06.533920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.533748    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-certs" pod "772fd287-d421-4a6e-86cb-c9ef846603df" (UID: "772fd287-d421-4a6e-86cb-c9ef846603df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:06.533920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.533790    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772fd287-d421-4a6e-86cb-c9ef846603df-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:06.534037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.533993    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-socket" pod "772fd287-d421-4a6e-86cb-c9ef846603df" (UID: "772fd287-d421-4a6e-86cb-c9ef846603df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:06.535243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.535215    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-istio-token" pod "772fd287-d421-4a6e-86cb-c9ef846603df" (UID: "772fd287-d421-4a6e-86cb-c9ef846603df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:41:06.535611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.535591    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-envoy" pod "772fd287-d421-4a6e-86cb-c9ef846603df" (UID: "772fd287-d421-4a6e-86cb-c9ef846603df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:41:06.535709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.535689    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-kube-api-access-x7pk4" pod "772fd287-d421-4a6e-86cb-c9ef846603df" (UID: "772fd287-d421-4a6e-86cb-c9ef846603df"). InnerVolumeSpecName "kube-api-access-x7pk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:41:06.535790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.535776    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/772fd287-d421-4a6e-86cb-c9ef846603df-istio-podinfo" pod "772fd287-d421-4a6e-86cb-c9ef846603df" (UID: "772fd287-d421-4a6e-86cb-c9ef846603df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:41:06.635505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.635367    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:06.635505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.635393    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:06.635505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.635402    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:06.635505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.635412    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7pk4\" (UniqueName: \"kubernetes.io/projected/772fd287-d421-4a6e-86cb-c9ef846603df-kube-api-access-x7pk4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:06.635505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.635422    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:06.635505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.635430    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772fd287-d421-4a6e-86cb-c9ef846603df-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:06.635505 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.635438    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772fd287-d421-4a6e-86cb-c9ef846603df-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:41:06.743402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.743366    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 21:41:06.743823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:06.743422    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 21:41:07.274944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:07.274913    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"
Nov 02 21:41:07.274944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:07.274931    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2" event={"ID":"772fd287-d421-4a6e-86cb-c9ef846603df","Type":"ContainerDied","Data":"8d297db23adb2e61eda32ad412fe71cda3ec0bfb98096d29df94970031261221"}
Nov 02 21:41:07.275187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:07.274990    4528 scope.go:122] "RemoveContainer" containerID="9756ca8039029b7db63320b9e768072739d204172e416a3aa0590b58accfa373"
Nov 02 21:41:07.442980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:07.442944    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"]
Nov 02 21:41:07.492989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:07.492940    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-ckpv2"]
Nov 02 21:41:07.743952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:07.743824    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 21:41:07.743952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:07.743883    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 21:41:08.748460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:08.748431    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:41:08.748964 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:08.748946    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:41:08.749513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:08.749494    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:41:08.939380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:08.939339    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" path="/var/lib/kubelet/pods/772fd287-d421-4a6e-86cb-c9ef846603df/volumes"
Nov 02 21:41:11.049398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:11.049367    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"]
Nov 02 21:41:11.058540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:11.058520    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95300606-t8bmv"]
Nov 02 21:41:12.919163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:12.919090    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887feeaa-e71a-440c-b213-f72977bbfeb5" path="/var/lib/kubelet/pods/887feeaa-e71a-440c-b213-f72977bbfeb5/volumes"
Nov 02 21:41:14.114295 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:41:14.114260    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:41:27.403137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:27.403092    4528 generic.go:356] "Generic (PLEG): container finished" podID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerID="6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f" exitCode=0
Nov 02 21:41:27.403619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:27.403138    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" event={"ID":"e733fbf4-28fd-4a07-8357-a372bc92d8b3","Type":"ContainerDied","Data":"6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f"}
Nov 02 21:41:27.403619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:27.403530    4528 scope.go:122] "RemoveContainer" containerID="6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f"
Nov 02 21:41:27.405132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:27.405086    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:41:27.405203 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:27.405162    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:41:27.405203 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:27.405196    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:41:28.430077 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:28.430036    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" event={"ID":"e733fbf4-28fd-4a07-8357-a372bc92d8b3","Type":"ContainerStarted","Data":"8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d"}
Nov 02 21:41:28.663463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:28.663431    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:41:28.664417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:28.664395    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 21:41:28.664475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:28.664444    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 21:41:28.827360 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:28.827315    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:41:29.663253 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:29.663222    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 21:41:29.663639 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:29.663272    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 21:41:30.662715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:30.662680    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 21:41:30.662908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:30.662734    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 21:41:31.662756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:31.662720    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 21:41:31.663168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:31.662775    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 21:41:32.663551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:32.663520    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 21:41:32.663933 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:32.663570    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 21:41:33.667398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:33.667360    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:41:33.668427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:33.668407    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:41:34.302523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:34.302484    4528 scope.go:122] "RemoveContainer" containerID="d617797cbc9762b5f1ec9ee97531ed9cf666614e73e943abc144d2ef73a97e83"
Nov 02 21:41:39.542022 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.541989    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"]
Nov 02 21:41:39.542661 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542645    4528 cpu_manager.go:414] "containerMap: removing container" podUID="91b46a41-5785-4ed1-ba52-eae364782587" containerName="payload-processing"
Nov 02 21:41:39.542720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542663    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b46a41-5785-4ed1-ba52-eae364782587" containerName="payload-processing"
Nov 02 21:41:39.542720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542676    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy"
Nov 02 21:41:39.542720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542681    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy"
Nov 02 21:41:39.542720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542699    4528 cpu_manager.go:414] "containerMap: removing container" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy"
Nov 02 21:41:39.542720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542704    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy"
Nov 02 21:41:39.542889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542724    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4951470f-4373-4bce-a1db-f95395dbc27e" containerName="payload-pre-processing"
Nov 02 21:41:39.542889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542729    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4951470f-4373-4bce-a1db-f95395dbc27e" containerName="payload-pre-processing"
Nov 02 21:41:39.542889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542826    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b46a41-5785-4ed1-ba52-eae364782587" containerName="payload-processing"
Nov 02 21:41:39.542889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542836    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e058f5-4c9d-41e7-a301-22e81d9270b2" containerName="istio-proxy"
Nov 02 21:41:39.542889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542844    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="772fd287-d421-4a6e-86cb-c9ef846603df" containerName="istio-proxy"
Nov 02 21:41:39.542889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.542853    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4951470f-4373-4bce-a1db-f95395dbc27e" containerName="payload-pre-processing"
Nov 02 21:41:39.547796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.547781    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:39.598832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.598802    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"]
Nov 02 21:41:39.643574 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.643544    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38362c6b-9f43-41e2-9174-deb14ea99a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4\" (UID: \"38362c6b-9f43-41e2-9174-deb14ea99a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:39.643729 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.643690    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsms\" (UniqueName: \"kubernetes.io/projected/38362c6b-9f43-41e2-9174-deb14ea99a4f-kube-api-access-kwsms\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4\" (UID: \"38362c6b-9f43-41e2-9174-deb14ea99a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:39.645189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.645169    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e12f99-dockercfg-7677n\""
Nov 02 21:41:39.645272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.645225    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e12f99\""
Nov 02 21:41:39.744377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.744346    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsms\" (UniqueName: \"kubernetes.io/projected/38362c6b-9f43-41e2-9174-deb14ea99a4f-kube-api-access-kwsms\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4\" (UID: \"38362c6b-9f43-41e2-9174-deb14ea99a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:39.744541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.744493    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38362c6b-9f43-41e2-9174-deb14ea99a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4\" (UID: \"38362c6b-9f43-41e2-9174-deb14ea99a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:39.745256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.745237    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38362c6b-9f43-41e2-9174-deb14ea99a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4\" (UID: \"38362c6b-9f43-41e2-9174-deb14ea99a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:39.801357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.801291    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsms\" (UniqueName: \"kubernetes.io/projected/38362c6b-9f43-41e2-9174-deb14ea99a4f-kube-api-access-kwsms\") pod \"payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4\" (UID: \"38362c6b-9f43-41e2-9174-deb14ea99a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:39.861537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:39.861510    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:40.230085 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:41:40.230055    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38362c6b_9f43_41e2_9174_deb14ea99a4f.slice/crio-71af9afd376eee6c4c722ac0c3a56930bc52cc4c719b27e22fd7d7ce411b2c4f WatchSource:0}: Error finding container 71af9afd376eee6c4c722ac0c3a56930bc52cc4c719b27e22fd7d7ce411b2c4f: Status 404 returned error can't find the container with id 71af9afd376eee6c4c722ac0c3a56930bc52cc4c719b27e22fd7d7ce411b2c4f
Nov 02 21:41:40.233940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.233914    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"]
Nov 02 21:41:40.328016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.327983    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"]
Nov 02 21:41:40.333424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.333399    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:40.352103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.352077    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779a4c9f-0b02-4c72-93e9-c66c28892202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k\" (UID: \"779a4c9f-0b02-4c72-93e9-c66c28892202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:40.352274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.352182    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfkp\" (UniqueName: \"kubernetes.io/projected/779a4c9f-0b02-4c72-93e9-c66c28892202-kube-api-access-fqfkp\") pod \"payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k\" (UID: \"779a4c9f-0b02-4c72-93e9-c66c28892202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:40.384485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.384451    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"]
Nov 02 21:41:40.453290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.453249    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779a4c9f-0b02-4c72-93e9-c66c28892202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k\" (UID: \"779a4c9f-0b02-4c72-93e9-c66c28892202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:40.453489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.453335    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfkp\" (UniqueName: \"kubernetes.io/projected/779a4c9f-0b02-4c72-93e9-c66c28892202-kube-api-access-fqfkp\") pod \"payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k\" (UID: \"779a4c9f-0b02-4c72-93e9-c66c28892202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:40.454032 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.454011    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779a4c9f-0b02-4c72-93e9-c66c28892202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k\" (UID: \"779a4c9f-0b02-4c72-93e9-c66c28892202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:40.507651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.507560    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4" event={"ID":"38362c6b-9f43-41e2-9174-deb14ea99a4f","Type":"ContainerStarted","Data":"5747033cc13927f766a865485764afea532d78a4a235d45b6b1fb19c2ce0b15b"}
Nov 02 21:41:40.507651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.507622    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4" event={"ID":"38362c6b-9f43-41e2-9174-deb14ea99a4f","Type":"ContainerStarted","Data":"71af9afd376eee6c4c722ac0c3a56930bc52cc4c719b27e22fd7d7ce411b2c4f"}
Nov 02 21:41:40.507651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.507649    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:40.510502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.510476    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfkp\" (UniqueName: \"kubernetes.io/projected/779a4c9f-0b02-4c72-93e9-c66c28892202-kube-api-access-fqfkp\") pod \"payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k\" (UID: \"779a4c9f-0b02-4c72-93e9-c66c28892202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:40.624352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.624299    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4" podStartSLOduration=1.624285641 podStartE2EDuration="1.624285641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:40.621179488 +0000 UTC m=+1987.884715806" watchObservedRunningTime="2026-11-02 21:41:40.624285641 +0000 UTC m=+1987.887821957"
Nov 02 21:41:40.649020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:40.648974    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:41.018265 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:41:41.018221    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779a4c9f_0b02_4c72_93e9_c66c28892202.slice/crio-a5ea0b866c12e4538ef2add9062c27b4d28f5c423403c65cf5872caba76dcfc8 WatchSource:0}: Error finding container a5ea0b866c12e4538ef2add9062c27b4d28f5c423403c65cf5872caba76dcfc8: Status 404 returned error can't find the container with id a5ea0b866c12e4538ef2add9062c27b4d28f5c423403c65cf5872caba76dcfc8
Nov 02 21:41:41.020746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:41.020718    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"]
Nov 02 21:41:41.516805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:41.516770    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k" event={"ID":"779a4c9f-0b02-4c72-93e9-c66c28892202","Type":"ContainerStarted","Data":"6d51bbb31191d0ffd172dae9c8655593dec8214f74649f9a01cd1cfadb505a5a"}
Nov 02 21:41:41.517018 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:41.516825    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k" event={"ID":"779a4c9f-0b02-4c72-93e9-c66c28892202","Type":"ContainerStarted","Data":"a5ea0b866c12e4538ef2add9062c27b4d28f5c423403c65cf5872caba76dcfc8"}
Nov 02 21:41:41.635608 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:41.635554    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k" podStartSLOduration=1.635540118 podStartE2EDuration="1.635540118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:41.632596115 +0000 UTC m=+1988.896132435" watchObservedRunningTime="2026-11-02 21:41:41.635540118 +0000 UTC m=+1988.899076434"
Nov 02 21:41:42.522982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:42.522947    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:44.217080 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:41:44.214343    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:41:44.458716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:44.458676    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:41:51.519014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:51.518986    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:41:53.531474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:53.531447    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:41:59.667241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:59.667206    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"]
Nov 02 21:41:59.779537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:41:59.779503    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"]
Nov 02 21:42:04.668182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:04.668144    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f" podUID="01170016-26dc-45ec-9397-4c4483b38d39" containerName="payload-processing" containerID="cri-o://74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603" gracePeriod=25
Nov 02 21:42:04.780386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:04.780344    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x" podUID="b10d1567-6ff3-4257-9635-55320b8f3b2f" containerName="payload-pre-processing" containerID="cri-o://bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc" gracePeriod=25
Nov 02 21:42:04.945201 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:04.945170    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:42:05.066304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.066278    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:42:05.124033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.124001    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01170016-26dc-45ec-9397-4c4483b38d39-kube-api-access-q7kzl\" (UniqueName: \"kubernetes.io/projected/01170016-26dc-45ec-9397-4c4483b38d39-kube-api-access-q7kzl\") pod \"01170016-26dc-45ec-9397-4c4483b38d39\" (UID: \"01170016-26dc-45ec-9397-4c4483b38d39\") "
Nov 02 21:42:05.124228 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.124214    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01170016-26dc-45ec-9397-4c4483b38d39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01170016-26dc-45ec-9397-4c4483b38d39-plugins-config-volume\") pod \"01170016-26dc-45ec-9397-4c4483b38d39\" (UID: \"01170016-26dc-45ec-9397-4c4483b38d39\") "
Nov 02 21:42:05.124576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.124551    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01170016-26dc-45ec-9397-4c4483b38d39-plugins-config-volume" pod "01170016-26dc-45ec-9397-4c4483b38d39" (UID: "01170016-26dc-45ec-9397-4c4483b38d39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:42:05.124683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.124660    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01170016-26dc-45ec-9397-4c4483b38d39-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:05.126068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.126050    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01170016-26dc-45ec-9397-4c4483b38d39-kube-api-access-q7kzl" pod "01170016-26dc-45ec-9397-4c4483b38d39" (UID: "01170016-26dc-45ec-9397-4c4483b38d39"). InnerVolumeSpecName "kube-api-access-q7kzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:42:05.225633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.225551    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10d1567-6ff3-4257-9635-55320b8f3b2f-kube-api-access-tn4vm\" (UniqueName: \"kubernetes.io/projected/b10d1567-6ff3-4257-9635-55320b8f3b2f-kube-api-access-tn4vm\") pod \"b10d1567-6ff3-4257-9635-55320b8f3b2f\" (UID: \"b10d1567-6ff3-4257-9635-55320b8f3b2f\") "
Nov 02 21:42:05.225633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.225609    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10d1567-6ff3-4257-9635-55320b8f3b2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10d1567-6ff3-4257-9635-55320b8f3b2f-plugins-config-volume\") pod \"b10d1567-6ff3-4257-9635-55320b8f3b2f\" (UID: \"b10d1567-6ff3-4257-9635-55320b8f3b2f\") "
Nov 02 21:42:05.226024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.225999    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10d1567-6ff3-4257-9635-55320b8f3b2f-plugins-config-volume" pod "b10d1567-6ff3-4257-9635-55320b8f3b2f" (UID: "b10d1567-6ff3-4257-9635-55320b8f3b2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:42:05.226145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.226133    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7kzl\" (UniqueName: \"kubernetes.io/projected/01170016-26dc-45ec-9397-4c4483b38d39-kube-api-access-q7kzl\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:05.226232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.226150    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10d1567-6ff3-4257-9635-55320b8f3b2f-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:05.227720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.227699    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10d1567-6ff3-4257-9635-55320b8f3b2f-kube-api-access-tn4vm" pod "b10d1567-6ff3-4257-9635-55320b8f3b2f" (UID: "b10d1567-6ff3-4257-9635-55320b8f3b2f"). InnerVolumeSpecName "kube-api-access-tn4vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:42:05.327405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.327371    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4vm\" (UniqueName: \"kubernetes.io/projected/b10d1567-6ff3-4257-9635-55320b8f3b2f-kube-api-access-tn4vm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:05.667363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.667326    4528 generic.go:356] "Generic (PLEG): container finished" podID="01170016-26dc-45ec-9397-4c4483b38d39" containerID="74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603" exitCode=0
Nov 02 21:42:05.667523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.667374    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"
Nov 02 21:42:05.667523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.667410    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f" event={"ID":"01170016-26dc-45ec-9397-4c4483b38d39","Type":"ContainerDied","Data":"74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603"}
Nov 02 21:42:05.667523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.667466    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f" event={"ID":"01170016-26dc-45ec-9397-4c4483b38d39","Type":"ContainerDied","Data":"bd6e1b4d7b2d20c237f4813354be2d7234cf190b42712dfa34cfc1a75c029e3e"}
Nov 02 21:42:05.667523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.667494    4528 scope.go:122] "RemoveContainer" containerID="74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603"
Nov 02 21:42:05.669087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.669061    4528 generic.go:356] "Generic (PLEG): container finished" podID="b10d1567-6ff3-4257-9635-55320b8f3b2f" containerID="bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc" exitCode=0
Nov 02 21:42:05.669454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.669098    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x" event={"ID":"b10d1567-6ff3-4257-9635-55320b8f3b2f","Type":"ContainerDied","Data":"bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc"}
Nov 02 21:42:05.669454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.669205    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x" event={"ID":"b10d1567-6ff3-4257-9635-55320b8f3b2f","Type":"ContainerDied","Data":"c6673bd4f29dfbaeb94d8a00f33c71259eefa04abb54419fd8cf996c9d4a0671"}
Nov 02 21:42:05.669454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.669138    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"
Nov 02 21:42:05.678405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.678250    4528 scope.go:122] "RemoveContainer" containerID="74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603"
Nov 02 21:42:05.678560 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:42:05.678541    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603\": container with ID starting with 74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603 not found: ID does not exist" containerID="74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603"
Nov 02 21:42:05.678606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.678566    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603"} err="failed to get container status \"74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603\": rpc error: code = NotFound desc = could not find container \"74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603\": container with ID starting with 74ce29b7a90a493e64e8afe0963289e20c22ad7f5b5155d4c87bd95b4bb19603 not found: ID does not exist"
Nov 02 21:42:05.678606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.678584    4528 scope.go:122] "RemoveContainer" containerID="bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc"
Nov 02 21:42:05.692206 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.692191    4528 scope.go:122] "RemoveContainer" containerID="bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc"
Nov 02 21:42:05.692449 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:42:05.692431    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc\": container with ID starting with bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc not found: ID does not exist" containerID="bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc"
Nov 02 21:42:05.692500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.692456    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc"} err="failed to get container status \"bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc\": rpc error: code = NotFound desc = could not find container \"bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc\": container with ID starting with bed69f19b4daa2c589f130de0566ac31da2ec1b814f806888c35685637ff62cc not found: ID does not exist"
Nov 02 21:42:05.835796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.835768    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"]
Nov 02 21:42:05.886055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.886026    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4453bd-77c77c844c-llw5x"]
Nov 02 21:42:05.947439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.947355    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"]
Nov 02 21:42:05.951175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:05.951148    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4453bd-74c856bd59-5b97f"]
Nov 02 21:42:06.944661 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:06.944624    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01170016-26dc-45ec-9397-4c4483b38d39" path="/var/lib/kubelet/pods/01170016-26dc-45ec-9397-4c4483b38d39/volumes"
Nov 02 21:42:06.947324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:06.947296    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10d1567-6ff3-4257-9635-55320b8f3b2f" path="/var/lib/kubelet/pods/b10d1567-6ff3-4257-9635-55320b8f3b2f/volumes"
Nov 02 21:42:14.112686 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:42:14.112652    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:42:32.521323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:32.521283    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:42:34.421302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.421262    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"]
Nov 02 21:42:34.422702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.422431    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn" podUID="29fd19f0-2a8c-4c26-8536-d02013df7a57" containerName="payload-processing" containerID="cri-o://2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52" gracePeriod=30
Nov 02 21:42:34.426621 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.426597    4528 scope.go:122] "RemoveContainer" containerID="edf0b023f763e074a26984363d71ac4dcc94d90bee55ef52ee408e16de9a9835"
Nov 02 21:42:34.450970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.450945    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"]
Nov 02 21:42:34.451161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.451140    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq" podUID="84e8162f-c535-478b-832c-53d433bf98af" containerName="payload-pre-processing" containerID="cri-o://65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb" gracePeriod=30
Nov 02 21:42:34.781767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.781737    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:42:34.830757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.827391    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fd19f0-2a8c-4c26-8536-d02013df7a57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fd19f0-2a8c-4c26-8536-d02013df7a57-plugins-config-volume\") pod \"29fd19f0-2a8c-4c26-8536-d02013df7a57\" (UID: \"29fd19f0-2a8c-4c26-8536-d02013df7a57\") "
Nov 02 21:42:34.830757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.827651    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fd19f0-2a8c-4c26-8536-d02013df7a57-kube-api-access-vqfcc\" (UniqueName: \"kubernetes.io/projected/29fd19f0-2a8c-4c26-8536-d02013df7a57-kube-api-access-vqfcc\") pod \"29fd19f0-2a8c-4c26-8536-d02013df7a57\" (UID: \"29fd19f0-2a8c-4c26-8536-d02013df7a57\") "
Nov 02 21:42:34.830757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.828531    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fd19f0-2a8c-4c26-8536-d02013df7a57-plugins-config-volume" pod "29fd19f0-2a8c-4c26-8536-d02013df7a57" (UID: "29fd19f0-2a8c-4c26-8536-d02013df7a57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:42:34.864272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.848778    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:42:34.864272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.850642    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fd19f0-2a8c-4c26-8536-d02013df7a57-kube-api-access-vqfcc" pod "29fd19f0-2a8c-4c26-8536-d02013df7a57" (UID: "29fd19f0-2a8c-4c26-8536-d02013df7a57"). InnerVolumeSpecName "kube-api-access-vqfcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:42:34.929197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.929157    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqfcc\" (UniqueName: \"kubernetes.io/projected/29fd19f0-2a8c-4c26-8536-d02013df7a57-kube-api-access-vqfcc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:34.929197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:34.929198    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29fd19f0-2a8c-4c26-8536-d02013df7a57-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:35.091275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.086543    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e8162f-c535-478b-832c-53d433bf98af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e8162f-c535-478b-832c-53d433bf98af-plugins-config-volume\") pod \"84e8162f-c535-478b-832c-53d433bf98af\" (UID: \"84e8162f-c535-478b-832c-53d433bf98af\") "
Nov 02 21:42:35.091275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.086676    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e8162f-c535-478b-832c-53d433bf98af-kube-api-access-gzt8x\" (UniqueName: \"kubernetes.io/projected/84e8162f-c535-478b-832c-53d433bf98af-kube-api-access-gzt8x\") pod \"84e8162f-c535-478b-832c-53d433bf98af\" (UID: \"84e8162f-c535-478b-832c-53d433bf98af\") "
Nov 02 21:42:35.091275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.088202    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e8162f-c535-478b-832c-53d433bf98af-plugins-config-volume" pod "84e8162f-c535-478b-832c-53d433bf98af" (UID: "84e8162f-c535-478b-832c-53d433bf98af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:42:35.170141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.169088    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e8162f-c535-478b-832c-53d433bf98af-kube-api-access-gzt8x" pod "84e8162f-c535-478b-832c-53d433bf98af" (UID: "84e8162f-c535-478b-832c-53d433bf98af"). InnerVolumeSpecName "kube-api-access-gzt8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:42:35.181379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.181340    4528 generic.go:356] "Generic (PLEG): container finished" podID="29fd19f0-2a8c-4c26-8536-d02013df7a57" containerID="2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52" exitCode=0
Nov 02 21:42:35.181561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.181378    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn" event={"ID":"29fd19f0-2a8c-4c26-8536-d02013df7a57","Type":"ContainerDied","Data":"2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52"}
Nov 02 21:42:35.181561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.181447    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn" event={"ID":"29fd19f0-2a8c-4c26-8536-d02013df7a57","Type":"ContainerDied","Data":"f83edb90d689f8103b5a2bc313b94eab2e035dd8adf4d0c9c8d405374e967ca2"}
Nov 02 21:42:35.181561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.181483    4528 scope.go:122] "RemoveContainer" containerID="2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52"
Nov 02 21:42:35.181561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.181549    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"
Nov 02 21:42:35.189083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.189054    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e8162f-c535-478b-832c-53d433bf98af-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:35.189384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.189367    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzt8x\" (UniqueName: \"kubernetes.io/projected/84e8162f-c535-478b-832c-53d433bf98af-kube-api-access-gzt8x\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:35.190283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.190258    4528 generic.go:356] "Generic (PLEG): container finished" podID="84e8162f-c535-478b-832c-53d433bf98af" containerID="65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb" exitCode=0
Nov 02 21:42:35.190459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.190442    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq" event={"ID":"84e8162f-c535-478b-832c-53d433bf98af","Type":"ContainerDied","Data":"65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb"}
Nov 02 21:42:35.190592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.190578    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq" event={"ID":"84e8162f-c535-478b-832c-53d433bf98af","Type":"ContainerDied","Data":"173ba9387b7bb18f707a95295784e8add875e458cc70a13f689a240eef1249ac"}
Nov 02 21:42:35.190771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.190751    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"
Nov 02 21:42:35.208362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.208343    4528 scope.go:122] "RemoveContainer" containerID="2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52"
Nov 02 21:42:35.211336 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:42:35.208689    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52\": container with ID starting with 2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52 not found: ID does not exist" containerID="2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52"
Nov 02 21:42:35.211336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.208724    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52"} err="failed to get container status \"2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52\": rpc error: code = NotFound desc = could not find container \"2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52\": container with ID starting with 2e8caa945fede4fed713e3f3ba604dc3d5baac360869a1cbae2cde2562a49a52 not found: ID does not exist"
Nov 02 21:42:35.211336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.208746    4528 scope.go:122] "RemoveContainer" containerID="65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb"
Nov 02 21:42:35.240417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.240396    4528 scope.go:122] "RemoveContainer" containerID="65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb"
Nov 02 21:42:35.240770 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:42:35.240751    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb\": container with ID starting with 65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb not found: ID does not exist" containerID="65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb"
Nov 02 21:42:35.240849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.240781    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb"} err="failed to get container status \"65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb\": rpc error: code = NotFound desc = could not find container \"65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb\": container with ID starting with 65857310cbb35f3a5585fff31f1ff06ee027dcac02150b94951684a98bdd10cb not found: ID does not exist"
Nov 02 21:42:35.349975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.349888    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"]
Nov 02 21:42:35.353938 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.353911    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54e844-7dfb7bffcf-kg2kq"]
Nov 02 21:42:35.470752 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.470710    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"]
Nov 02 21:42:35.473563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:35.473544    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54e844-57f9f587d-j5zvn"]
Nov 02 21:42:36.932913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:36.932878    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fd19f0-2a8c-4c26-8536-d02013df7a57" path="/var/lib/kubelet/pods/29fd19f0-2a8c-4c26-8536-d02013df7a57/volumes"
Nov 02 21:42:36.933346 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:36.933327    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e8162f-c535-478b-832c-53d433bf98af" path="/var/lib/kubelet/pods/84e8162f-c535-478b-832c-53d433bf98af/volumes"
Nov 02 21:42:39.098517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:39.098480    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"]
Nov 02 21:42:39.204690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:39.204609    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"]
Nov 02 21:42:42.333257 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:42.333221    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:42:42.335649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:42.333313    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:42:43.286479 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:43.286439    4528 generic.go:356] "Generic (PLEG): container finished" podID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerID="2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06" exitCode=0
Nov 02 21:42:43.286656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:43.286507    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" event={"ID":"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef","Type":"ContainerDied","Data":"2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06"}
Nov 02 21:42:43.286958 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:43.286940    4528 scope.go:122] "RemoveContainer" containerID="2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06"
Nov 02 21:42:43.288488 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:43.288461    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:42:43.288560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:43.288526    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:42:43.288560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:43.288555    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:42:44.099759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.099713    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs" podUID="b0cbde66-94cd-4205-b99a-197f1b14c84d" containerName="payload-processing" containerID="cri-o://8d9f0e9b0e21406a08e56b2eb674623949528d344db8940cd8f6d7491f4b0769" gracePeriod=25
Nov 02 21:42:44.114041 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:42:44.114001    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:42:44.205548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.205500    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55" podUID="ccd3ccbb-c199-4540-8e31-0e7c53f44027" containerName="payload-pre-processing" containerID="cri-o://3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24" gracePeriod=25
Nov 02 21:42:44.299332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.298698    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" event={"ID":"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef","Type":"ContainerStarted","Data":"439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178"}
Nov 02 21:42:44.301251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.301223    4528 generic.go:356] "Generic (PLEG): container finished" podID="b0cbde66-94cd-4205-b99a-197f1b14c84d" containerID="8d9f0e9b0e21406a08e56b2eb674623949528d344db8940cd8f6d7491f4b0769" exitCode=0
Nov 02 21:42:44.301398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.301330    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs" event={"ID":"b0cbde66-94cd-4205-b99a-197f1b14c84d","Type":"ContainerDied","Data":"8d9f0e9b0e21406a08e56b2eb674623949528d344db8940cd8f6d7491f4b0769"}
Nov 02 21:42:44.368438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.368379    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:42:44.368611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.368584    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 21:42:44.368687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.368628    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 21:42:44.372217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.372191    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:42:44.430129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.430085    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cbde66-94cd-4205-b99a-197f1b14c84d-kube-api-access-5ccbh\" (UniqueName: \"kubernetes.io/projected/b0cbde66-94cd-4205-b99a-197f1b14c84d-kube-api-access-5ccbh\") pod \"b0cbde66-94cd-4205-b99a-197f1b14c84d\" (UID: \"b0cbde66-94cd-4205-b99a-197f1b14c84d\") "
Nov 02 21:42:44.430327 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.430236    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cbde66-94cd-4205-b99a-197f1b14c84d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbde66-94cd-4205-b99a-197f1b14c84d-plugins-config-volume\") pod \"b0cbde66-94cd-4205-b99a-197f1b14c84d\" (UID: \"b0cbde66-94cd-4205-b99a-197f1b14c84d\") "
Nov 02 21:42:44.430886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.430856    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cbde66-94cd-4205-b99a-197f1b14c84d-plugins-config-volume" pod "b0cbde66-94cd-4205-b99a-197f1b14c84d" (UID: "b0cbde66-94cd-4205-b99a-197f1b14c84d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:42:44.436934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.436903    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cbde66-94cd-4205-b99a-197f1b14c84d-kube-api-access-5ccbh" pod "b0cbde66-94cd-4205-b99a-197f1b14c84d" (UID: "b0cbde66-94cd-4205-b99a-197f1b14c84d"). InnerVolumeSpecName "kube-api-access-5ccbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:42:44.477809 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.477786    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:42:44.531482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.531453    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd3ccbb-c199-4540-8e31-0e7c53f44027-kube-api-access-clgzc\" (UniqueName: \"kubernetes.io/projected/ccd3ccbb-c199-4540-8e31-0e7c53f44027-kube-api-access-clgzc\") pod \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\" (UID: \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\") "
Nov 02 21:42:44.531627 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.531608    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd3ccbb-c199-4540-8e31-0e7c53f44027-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd3ccbb-c199-4540-8e31-0e7c53f44027-plugins-config-volume\") pod \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\" (UID: \"ccd3ccbb-c199-4540-8e31-0e7c53f44027\") "
Nov 02 21:42:44.532005 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.531982    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd3ccbb-c199-4540-8e31-0e7c53f44027-plugins-config-volume" pod "ccd3ccbb-c199-4540-8e31-0e7c53f44027" (UID: "ccd3ccbb-c199-4540-8e31-0e7c53f44027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:42:44.532486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.532467    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd3ccbb-c199-4540-8e31-0e7c53f44027-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:44.532573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.532487    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ccbh\" (UniqueName: \"kubernetes.io/projected/b0cbde66-94cd-4205-b99a-197f1b14c84d-kube-api-access-5ccbh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:44.532573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.532498    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cbde66-94cd-4205-b99a-197f1b14c84d-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:44.533629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.533611    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd3ccbb-c199-4540-8e31-0e7c53f44027-kube-api-access-clgzc" pod "ccd3ccbb-c199-4540-8e31-0e7c53f44027" (UID: "ccd3ccbb-c199-4540-8e31-0e7c53f44027"). InnerVolumeSpecName "kube-api-access-clgzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:42:44.633761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:44.633683    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clgzc\" (UniqueName: \"kubernetes.io/projected/ccd3ccbb-c199-4540-8e31-0e7c53f44027-kube-api-access-clgzc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:42:45.329703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.329667    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"
Nov 02 21:42:45.330344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.329677    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs" event={"ID":"b0cbde66-94cd-4205-b99a-197f1b14c84d","Type":"ContainerDied","Data":"c0147f13da40d7afee05b22c44de2c1451d8c05c522051bb3d0fde166dc646e8"}
Nov 02 21:42:45.330344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.330216    4528 scope.go:122] "RemoveContainer" containerID="8d9f0e9b0e21406a08e56b2eb674623949528d344db8940cd8f6d7491f4b0769"
Nov 02 21:42:45.331667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.331251    4528 generic.go:356] "Generic (PLEG): container finished" podID="ccd3ccbb-c199-4540-8e31-0e7c53f44027" containerID="3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24" exitCode=0
Nov 02 21:42:45.331667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.331313    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55" event={"ID":"ccd3ccbb-c199-4540-8e31-0e7c53f44027","Type":"ContainerDied","Data":"3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24"}
Nov 02 21:42:45.331667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.331379    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55" event={"ID":"ccd3ccbb-c199-4540-8e31-0e7c53f44027","Type":"ContainerDied","Data":"fb8ede247c3943968e8889dc158500b657bc1a962793c24a75ea27412bb454f0"}
Nov 02 21:42:45.331667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.331326    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"
Nov 02 21:42:45.339866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.339847    4528 scope.go:122] "RemoveContainer" containerID="3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24"
Nov 02 21:42:45.350582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.350558    4528 scope.go:122] "RemoveContainer" containerID="3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24"
Nov 02 21:42:45.350841 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:42:45.350814    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24\": container with ID starting with 3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24 not found: ID does not exist" containerID="3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24"
Nov 02 21:42:45.350927 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.350852    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24"} err="failed to get container status \"3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24\": rpc error: code = NotFound desc = could not find container \"3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24\": container with ID starting with 3f084582ca0d7aa61d6a4b25cae011bfda2e5bcd7b22c4cfdc732a4e9ea98e24 not found: ID does not exist"
Nov 02 21:42:45.369537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.369509    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 21:42:45.369630 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.369557    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 21:42:45.495234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.495200    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"]
Nov 02 21:42:45.497881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.497858    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-78db9bc887-g7j55"]
Nov 02 21:42:45.628733 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.628631    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"]
Nov 02 21:42:45.631947 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:45.631919    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-7c78c7ddd7-8bzjs"]
Nov 02 21:42:46.368923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:46.368891    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 21:42:46.369378 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:46.368946    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 21:42:46.934968 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:46.934931    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cbde66-94cd-4205-b99a-197f1b14c84d" path="/var/lib/kubelet/pods/b0cbde66-94cd-4205-b99a-197f1b14c84d/volumes"
Nov 02 21:42:46.935496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:46.935473    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd3ccbb-c199-4540-8e31-0e7c53f44027" path="/var/lib/kubelet/pods/ccd3ccbb-c199-4540-8e31-0e7c53f44027/volumes"
Nov 02 21:42:47.369096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:47.369061    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 21:42:47.369503 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:47.369147    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 21:42:48.368802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:48.368774    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 21:42:48.369061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:48.369039    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 21:42:49.369225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:49.369197    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 21:42:49.369626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:49.369246    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 21:42:50.407999 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:50.407963    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:42:50.408468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:50.408366    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:42:50.408996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:42:50.408977    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:43:10.219356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:10.219319    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"]
Nov 02 21:43:10.219881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:10.219589    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy" containerID="cri-o://1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b" gracePeriod=30
Nov 02 21:43:10.222143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:10.222104    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"]
Nov 02 21:43:14.113005 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:43:14.112952    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:43:15.222760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.222693    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f" podUID="be25b2b4-327d-46f0-b23e-72b71ed06af0" containerName="payload-pre-processing" containerID="cri-o://ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0" gracePeriod=25
Nov 02 21:43:15.484186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.484093    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:43:15.530538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.530515    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:43:15.548418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.548387    4528 generic.go:356] "Generic (PLEG): container finished" podID="be25b2b4-327d-46f0-b23e-72b71ed06af0" containerID="ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0" exitCode=0
Nov 02 21:43:15.548540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.548427    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"
Nov 02 21:43:15.548540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.548466    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f" event={"ID":"be25b2b4-327d-46f0-b23e-72b71ed06af0","Type":"ContainerDied","Data":"ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0"}
Nov 02 21:43:15.548540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.548512    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f" event={"ID":"be25b2b4-327d-46f0-b23e-72b71ed06af0","Type":"ContainerDied","Data":"04a9d32953a3c265799b8560358b337154431783297b82638f0ea15252c868a6"}
Nov 02 21:43:15.548540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.548537    4528 scope.go:122] "RemoveContainer" containerID="ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0"
Nov 02 21:43:15.550097 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.550078    4528 generic.go:356] "Generic (PLEG): container finished" podID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerID="1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b" exitCode=0
Nov 02 21:43:15.550240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.550131    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" event={"ID":"a7af38e3-4322-46b6-9eb6-638c9a3e5be0","Type":"ContainerDied","Data":"1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b"}
Nov 02 21:43:15.550240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.550157    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"
Nov 02 21:43:15.550240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.550173    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd" event={"ID":"a7af38e3-4322-46b6-9eb6-638c9a3e5be0","Type":"ContainerDied","Data":"bf95a35340c2fc4668be1fd3dc8a2089f30d460bbb8a439b68bc5a069bc96cb8"}
Nov 02 21:43:15.557818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.557800    4528 scope.go:122] "RemoveContainer" containerID="ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0"
Nov 02 21:43:15.558071 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:43:15.558055    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0\": container with ID starting with ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0 not found: ID does not exist" containerID="ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0"
Nov 02 21:43:15.558159 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.558078    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0"} err="failed to get container status \"ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0\": rpc error: code = NotFound desc = could not find container \"ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0\": container with ID starting with ec2a065fec1b5bd58188658f8e7d1c69fd73b6222ec8afa239febab05fea4bb0 not found: ID does not exist"
Nov 02 21:43:15.558159 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.558092    4528 scope.go:122] "RemoveContainer" containerID="1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b"
Nov 02 21:43:15.566951 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.566934    4528 scope.go:122] "RemoveContainer" containerID="f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8"
Nov 02 21:43:15.575414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.575395    4528 scope.go:122] "RemoveContainer" containerID="1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b"
Nov 02 21:43:15.575675 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:43:15.575657    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b\": container with ID starting with 1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b not found: ID does not exist" containerID="1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b"
Nov 02 21:43:15.575722 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.575684    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b"} err="failed to get container status \"1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b\": rpc error: code = NotFound desc = could not find container \"1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b\": container with ID starting with 1bd6defa8124b62438a0adc89140b0e31a6289f11067ba2f4946b23ed63cbe7b not found: ID does not exist"
Nov 02 21:43:15.575722 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.575700    4528 scope.go:122] "RemoveContainer" containerID="f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8"
Nov 02 21:43:15.575947 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:43:15.575933    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8\": container with ID starting with f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8 not found: ID does not exist" containerID="f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8"
Nov 02 21:43:15.575996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.575951    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8"} err="failed to get container status \"f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8\": rpc error: code = NotFound desc = could not find container \"f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8\": container with ID starting with f0097872cbccb06da4edbdf1159a2724f9990fabb4bb92aadfb714b050784cc8 not found: ID does not exist"
Nov 02 21:43:15.579362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.579341    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be25b2b4-327d-46f0-b23e-72b71ed06af0-kube-api-access-c9h6q\" (UniqueName: \"kubernetes.io/projected/be25b2b4-327d-46f0-b23e-72b71ed06af0-kube-api-access-c9h6q\") pod \"be25b2b4-327d-46f0-b23e-72b71ed06af0\" (UID: \"be25b2b4-327d-46f0-b23e-72b71ed06af0\") "
Nov 02 21:43:15.579454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.579375    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be25b2b4-327d-46f0-b23e-72b71ed06af0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be25b2b4-327d-46f0-b23e-72b71ed06af0-plugins-config-volume\") pod \"be25b2b4-327d-46f0-b23e-72b71ed06af0\" (UID: \"be25b2b4-327d-46f0-b23e-72b71ed06af0\") "
Nov 02 21:43:15.579785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.579765    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be25b2b4-327d-46f0-b23e-72b71ed06af0-plugins-config-volume" pod "be25b2b4-327d-46f0-b23e-72b71ed06af0" (UID: "be25b2b4-327d-46f0-b23e-72b71ed06af0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:43:15.579991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.579978    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be25b2b4-327d-46f0-b23e-72b71ed06af0-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.581379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.581358    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be25b2b4-327d-46f0-b23e-72b71ed06af0-kube-api-access-c9h6q" pod "be25b2b4-327d-46f0-b23e-72b71ed06af0" (UID: "be25b2b4-327d-46f0-b23e-72b71ed06af0"). InnerVolumeSpecName "kube-api-access-c9h6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:43:15.680534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680505    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-kube-api-access-7d7lh\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-kube-api-access-7d7lh\") pod \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") "
Nov 02 21:43:15.680718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680551    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-credential-socket\") pod \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") "
Nov 02 21:43:15.680718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680573    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-podinfo\") pod \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") "
Nov 02 21:43:15.680718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680607    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-certs\") pod \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") "
Nov 02 21:43:15.680718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680672    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-data\") pod \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") "
Nov 02 21:43:15.680956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680759    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-socket\") pod \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") "
Nov 02 21:43:15.680956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680802    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-token\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-token\") pod \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") "
Nov 02 21:43:15.680956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680801    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-credential-socket" pod "a7af38e3-4322-46b6-9eb6-638c9a3e5be0" (UID: "a7af38e3-4322-46b6-9eb6-638c9a3e5be0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:15.680956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680837    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istiod-ca-cert\") pod \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") "
Nov 02 21:43:15.680956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.680903    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-envoy\") pod \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\" (UID: \"a7af38e3-4322-46b6-9eb6-638c9a3e5be0\") "
Nov 02 21:43:15.681278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681009    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-certs" pod "a7af38e3-4322-46b6-9eb6-638c9a3e5be0" (UID: "a7af38e3-4322-46b6-9eb6-638c9a3e5be0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:15.681278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681037    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-data" pod "a7af38e3-4322-46b6-9eb6-638c9a3e5be0" (UID: "a7af38e3-4322-46b6-9eb6-638c9a3e5be0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:15.681278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681099    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-socket" pod "a7af38e3-4322-46b6-9eb6-638c9a3e5be0" (UID: "a7af38e3-4322-46b6-9eb6-638c9a3e5be0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:15.681444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681412    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istiod-ca-cert" pod "a7af38e3-4322-46b6-9eb6-638c9a3e5be0" (UID: "a7af38e3-4322-46b6-9eb6-638c9a3e5be0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:43:15.681549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681530    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.681615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681554    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.681615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681569    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.681615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681583    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.681615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681598    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.681615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.681612    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9h6q\" (UniqueName: \"kubernetes.io/projected/be25b2b4-327d-46f0-b23e-72b71ed06af0-kube-api-access-c9h6q\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.682987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.682959    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-kube-api-access-7d7lh" pod "a7af38e3-4322-46b6-9eb6-638c9a3e5be0" (UID: "a7af38e3-4322-46b6-9eb6-638c9a3e5be0"). InnerVolumeSpecName "kube-api-access-7d7lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:43:15.682987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.682963    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-podinfo" pod "a7af38e3-4322-46b6-9eb6-638c9a3e5be0" (UID: "a7af38e3-4322-46b6-9eb6-638c9a3e5be0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:43:15.683151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.682992    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-envoy" pod "a7af38e3-4322-46b6-9eb6-638c9a3e5be0" (UID: "a7af38e3-4322-46b6-9eb6-638c9a3e5be0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:15.683151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.683024    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-token" pod "a7af38e3-4322-46b6-9eb6-638c9a3e5be0" (UID: "a7af38e3-4322-46b6-9eb6-638c9a3e5be0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:43:15.782802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.782771    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.782802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.782799    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.782996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.782809    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7lh\" (UniqueName: \"kubernetes.io/projected/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-kube-api-access-7d7lh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.782996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.782820    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7af38e3-4322-46b6-9eb6-638c9a3e5be0-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:15.972278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.972241    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"]
Nov 02 21:43:15.974919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:15.974898    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-54454cc47-8vw4f"]
Nov 02 21:43:16.085287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:16.085202    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"]
Nov 02 21:43:16.087838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:16.087815    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4453bd-openshift-default-c68dfc76f-r4ngd"]
Nov 02 21:43:16.914101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:16.914059    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" path="/var/lib/kubelet/pods/a7af38e3-4322-46b6-9eb6-638c9a3e5be0/volumes"
Nov 02 21:43:16.914678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:16.914657    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be25b2b4-327d-46f0-b23e-72b71ed06af0" path="/var/lib/kubelet/pods/be25b2b4-327d-46f0-b23e-72b71ed06af0/volumes"
Nov 02 21:43:17.194652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:17.194579    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"]
Nov 02 21:43:18.999191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:18.999159    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:43:21.245502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.245471    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"]
Nov 02 21:43:21.246165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246144    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ccd3ccbb-c199-4540-8e31-0e7c53f44027" containerName="payload-pre-processing"
Nov 02 21:43:21.246283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246168    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd3ccbb-c199-4540-8e31-0e7c53f44027" containerName="payload-pre-processing"
Nov 02 21:43:21.246283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246204    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b10d1567-6ff3-4257-9635-55320b8f3b2f" containerName="payload-pre-processing"
Nov 02 21:43:21.246283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246212    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10d1567-6ff3-4257-9635-55320b8f3b2f" containerName="payload-pre-processing"
Nov 02 21:43:21.246283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246222    4528 cpu_manager.go:414] "containerMap: removing container" podUID="be25b2b4-327d-46f0-b23e-72b71ed06af0" containerName="payload-pre-processing"
Nov 02 21:43:21.246283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246230    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be25b2b4-327d-46f0-b23e-72b71ed06af0" containerName="payload-pre-processing"
Nov 02 21:43:21.246283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246251    4528 cpu_manager.go:414] "containerMap: removing container" podUID="84e8162f-c535-478b-832c-53d433bf98af" containerName="payload-pre-processing"
Nov 02 21:43:21.246283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246258    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e8162f-c535-478b-832c-53d433bf98af" containerName="payload-pre-processing"
Nov 02 21:43:21.246283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246273    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b0cbde66-94cd-4205-b99a-197f1b14c84d" containerName="payload-processing"
Nov 02 21:43:21.246283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246281    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cbde66-94cd-4205-b99a-197f1b14c84d" containerName="payload-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246296    4528 cpu_manager.go:414] "containerMap: removing container" podUID="01170016-26dc-45ec-9397-4c4483b38d39" containerName="payload-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246304    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01170016-26dc-45ec-9397-4c4483b38d39" containerName="payload-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246320    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246328    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246349    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246356    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246371    4528 cpu_manager.go:414] "containerMap: removing container" podUID="29fd19f0-2a8c-4c26-8536-d02013df7a57" containerName="payload-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246378    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fd19f0-2a8c-4c26-8536-d02013df7a57" containerName="payload-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246487    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd3ccbb-c199-4540-8e31-0e7c53f44027" containerName="payload-pre-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246502    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10d1567-6ff3-4257-9635-55320b8f3b2f" containerName="payload-pre-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246511    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="be25b2b4-327d-46f0-b23e-72b71ed06af0" containerName="payload-pre-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246522    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e8162f-c535-478b-832c-53d433bf98af" containerName="payload-pre-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246530    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cbde66-94cd-4205-b99a-197f1b14c84d" containerName="payload-processing"
Nov 02 21:43:21.246544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246539    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="01170016-26dc-45ec-9397-4c4483b38d39" containerName="payload-processing"
Nov 02 21:43:21.246938 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246548    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy"
Nov 02 21:43:21.246938 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246558    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7af38e3-4322-46b6-9eb6-638c9a3e5be0" containerName="istio-proxy"
Nov 02 21:43:21.246938 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.246568    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fd19f0-2a8c-4c26-8536-d02013df7a57" containerName="payload-processing"
Nov 02 21:43:21.251748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.251729    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:21.259288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.259264    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmtn\" (UniqueName: \"kubernetes.io/projected/bb43cdb9-4948-4b75-87a3-03151929ef51-kube-api-access-wfmtn\") pod \"payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw\" (UID: \"bb43cdb9-4948-4b75-87a3-03151929ef51\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:21.259373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.259318    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43cdb9-4948-4b75-87a3-03151929ef51-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw\" (UID: \"bb43cdb9-4948-4b75-87a3-03151929ef51\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:21.382546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.382500    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmtn\" (UniqueName: \"kubernetes.io/projected/bb43cdb9-4948-4b75-87a3-03151929ef51-kube-api-access-wfmtn\") pod \"payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw\" (UID: \"bb43cdb9-4948-4b75-87a3-03151929ef51\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:21.387338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.387305    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43cdb9-4948-4b75-87a3-03151929ef51-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw\" (UID: \"bb43cdb9-4948-4b75-87a3-03151929ef51\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:21.388452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.384737    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-be6df0-dockercfg-2mx7x\""
Nov 02 21:43:21.388589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.386755    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"]
Nov 02 21:43:21.388738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.384436    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-be6df0\""
Nov 02 21:43:21.388738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.388276    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43cdb9-4948-4b75-87a3-03151929ef51-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw\" (UID: \"bb43cdb9-4948-4b75-87a3-03151929ef51\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:21.438458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.438422    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"]
Nov 02 21:43:21.442975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.442955    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:21.444530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.444507    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmtn\" (UniqueName: \"kubernetes.io/projected/bb43cdb9-4948-4b75-87a3-03151929ef51-kube-api-access-wfmtn\") pod \"payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw\" (UID: \"bb43cdb9-4948-4b75-87a3-03151929ef51\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:21.467988 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.467963    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"]
Nov 02 21:43:21.489527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.489498    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdcck\" (UniqueName: \"kubernetes.io/projected/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-kube-api-access-gdcck\") pod \"payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29\" (UID: \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:21.489656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.489586    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29\" (UID: \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:21.565869 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.565846    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:21.590999 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.590966    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdcck\" (UniqueName: \"kubernetes.io/projected/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-kube-api-access-gdcck\") pod \"payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29\" (UID: \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:21.591127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.591078    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29\" (UID: \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:21.591744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.591716    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29\" (UID: \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:21.649327 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.649297    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdcck\" (UniqueName: \"kubernetes.io/projected/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-kube-api-access-gdcck\") pod \"payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29\" (UID: \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:21.769073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.769040    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:21.935707 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:43:21.935677    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb43cdb9_4948_4b75_87a3_03151929ef51.slice/crio-a5b775ad225f0d2655af6a92276dc27182bd12f6b43ba7a5163639cebcc57762 WatchSource:0}: Error finding container a5b775ad225f0d2655af6a92276dc27182bd12f6b43ba7a5163639cebcc57762: Status 404 returned error can't find the container with id a5b775ad225f0d2655af6a92276dc27182bd12f6b43ba7a5163639cebcc57762
Nov 02 21:43:21.938848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:21.938827    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"]
Nov 02 21:43:22.136683 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:43:22.136629    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e98f242_a7ba_4ddb_bb03_9f740e1670f1.slice/crio-3148f4577dcbc74158b80059502014a7c17ee3a36f1b58d2d8e04910d07cea21 WatchSource:0}: Error finding container 3148f4577dcbc74158b80059502014a7c17ee3a36f1b58d2d8e04910d07cea21: Status 404 returned error can't find the container with id 3148f4577dcbc74158b80059502014a7c17ee3a36f1b58d2d8e04910d07cea21
Nov 02 21:43:22.139419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.139390    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"]
Nov 02 21:43:22.194975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.194938    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv" podUID="ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7" containerName="payload-processing" containerID="cri-o://c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75" gracePeriod=25
Nov 02 21:43:22.437178 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.437155    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:43:22.501275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.501252    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-plugins-config-volume\") pod \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\" (UID: \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\") "
Nov 02 21:43:22.501425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.501414    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-kube-api-access-sxhgr\" (UniqueName: \"kubernetes.io/projected/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-kube-api-access-sxhgr\") pod \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\" (UID: \"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7\") "
Nov 02 21:43:22.501609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.501588    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-plugins-config-volume" pod "ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7" (UID: "ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:43:22.501762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.501742    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:22.503308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.503289    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-kube-api-access-sxhgr" pod "ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7" (UID: "ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7"). InnerVolumeSpecName "kube-api-access-sxhgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:43:22.602499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.602466    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxhgr\" (UniqueName: \"kubernetes.io/projected/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7-kube-api-access-sxhgr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:22.644621 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.644588    4528 generic.go:356] "Generic (PLEG): container finished" podID="ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7" containerID="c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75" exitCode=0
Nov 02 21:43:22.644785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.644635    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"
Nov 02 21:43:22.644785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.644667    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv" event={"ID":"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7","Type":"ContainerDied","Data":"c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75"}
Nov 02 21:43:22.644785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.644720    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv" event={"ID":"ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7","Type":"ContainerDied","Data":"d50aad885b7d3cf8b84b76da12d85915839c1ae0541bdfac36d7466e44a2a800"}
Nov 02 21:43:22.644785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.644746    4528 scope.go:122] "RemoveContainer" containerID="c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75"
Nov 02 21:43:22.646562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.646539    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw" event={"ID":"bb43cdb9-4948-4b75-87a3-03151929ef51","Type":"ContainerStarted","Data":"65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1"}
Nov 02 21:43:22.646669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.646588    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw" event={"ID":"bb43cdb9-4948-4b75-87a3-03151929ef51","Type":"ContainerStarted","Data":"a5b775ad225f0d2655af6a92276dc27182bd12f6b43ba7a5163639cebcc57762"}
Nov 02 21:43:22.646669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.646615    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:22.648215 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.648183    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29" event={"ID":"1e98f242-a7ba-4ddb-bb03-9f740e1670f1","Type":"ContainerStarted","Data":"8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea"}
Nov 02 21:43:22.648339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.648221    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29" event={"ID":"1e98f242-a7ba-4ddb-bb03-9f740e1670f1","Type":"ContainerStarted","Data":"3148f4577dcbc74158b80059502014a7c17ee3a36f1b58d2d8e04910d07cea21"}
Nov 02 21:43:22.648339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.648243    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:22.655214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.655195    4528 scope.go:122] "RemoveContainer" containerID="c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75"
Nov 02 21:43:22.655458 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:43:22.655441    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75\": container with ID starting with c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75 not found: ID does not exist" containerID="c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75"
Nov 02 21:43:22.655510 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.655465    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75"} err="failed to get container status \"c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75\": rpc error: code = NotFound desc = could not find container \"c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75\": container with ID starting with c50a0385fb977d813fb3ebb1eaaa33054d6e9b71148a52c58079eaf87e6f0a75 not found: ID does not exist"
Nov 02 21:43:22.814385 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.814336    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw" podStartSLOduration=1.814322378 podStartE2EDuration="1.814322378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:22.81288998 +0000 UTC m=+2090.076426302" watchObservedRunningTime="2026-11-02 21:43:22.814322378 +0000 UTC m=+2090.077858694"
Nov 02 21:43:22.930703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:22.930654    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29" podStartSLOduration=1.9306405500000001 podStartE2EDuration="1.93064055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:22.927792008 +0000 UTC m=+2090.191328328" watchObservedRunningTime="2026-11-02 21:43:22.93064055 +0000 UTC m=+2090.194176867"
Nov 02 21:43:23.040086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:23.040055    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"]
Nov 02 21:43:23.043936 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:23.043907    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-c9694c7cf-wnzjv"]
Nov 02 21:43:24.935133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:24.929767    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7" path="/var/lib/kubelet/pods/ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7/volumes"
Nov 02 21:43:33.658025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:33.657989    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:43:33.658493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:33.658134    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:43:34.852795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:34.852723    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pwkmv_53e705a3-6036-4fdd-800c-90b4eb36f5a8/payload-processing/0.log"
Nov 02 21:43:35.102933 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:35.102864    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pwkmv_53e705a3-6036-4fdd-800c-90b4eb36f5a8/payload-processing/0.log"
Nov 02 21:43:43.813631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:43.813601    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"]
Nov 02 21:43:43.814071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:43.813860    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy" containerID="cri-o://bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4" gracePeriod=30
Nov 02 21:43:44.112139 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:43:44.112035    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:43:47.988283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:47.988247    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:43:47.988650 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:47.988311    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:43:49.077838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.077781    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:43:49.137537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137506    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-podinfo\") pod \"49e7b2fb-8608-4235-aea2-b60c1e295270\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") "
Nov 02 21:43:49.137537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137541    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-credential-socket\") pod \"49e7b2fb-8608-4235-aea2-b60c1e295270\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") "
Nov 02 21:43:49.137774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137581    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-envoy\") pod \"49e7b2fb-8608-4235-aea2-b60c1e295270\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") "
Nov 02 21:43:49.137774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137609    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-socket\") pod \"49e7b2fb-8608-4235-aea2-b60c1e295270\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") "
Nov 02 21:43:49.137774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137663    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-kube-api-access-4n28j\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-kube-api-access-4n28j\") pod \"49e7b2fb-8608-4235-aea2-b60c1e295270\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") "
Nov 02 21:43:49.137774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137728    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-certs\") pod \"49e7b2fb-8608-4235-aea2-b60c1e295270\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") "
Nov 02 21:43:49.137774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137764    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-data\") pod \"49e7b2fb-8608-4235-aea2-b60c1e295270\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") "
Nov 02 21:43:49.138024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137849    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-token\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-token\") pod \"49e7b2fb-8608-4235-aea2-b60c1e295270\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") "
Nov 02 21:43:49.138024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137872    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-credential-socket" pod "49e7b2fb-8608-4235-aea2-b60c1e295270" (UID: "49e7b2fb-8608-4235-aea2-b60c1e295270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:49.138024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137901    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e7b2fb-8608-4235-aea2-b60c1e295270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e7b2fb-8608-4235-aea2-b60c1e295270-istiod-ca-cert\") pod \"49e7b2fb-8608-4235-aea2-b60c1e295270\" (UID: \"49e7b2fb-8608-4235-aea2-b60c1e295270\") "
Nov 02 21:43:49.138024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.137957    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-socket" pod "49e7b2fb-8608-4235-aea2-b60c1e295270" (UID: "49e7b2fb-8608-4235-aea2-b60c1e295270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:49.138286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.138258    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-data" pod "49e7b2fb-8608-4235-aea2-b60c1e295270" (UID: "49e7b2fb-8608-4235-aea2-b60c1e295270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:49.138334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.138278    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e7b2fb-8608-4235-aea2-b60c1e295270-istiod-ca-cert" pod "49e7b2fb-8608-4235-aea2-b60c1e295270" (UID: "49e7b2fb-8608-4235-aea2-b60c1e295270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:43:49.138389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.138331    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-certs" pod "49e7b2fb-8608-4235-aea2-b60c1e295270" (UID: "49e7b2fb-8608-4235-aea2-b60c1e295270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:49.138540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.138521    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:49.138577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.138548    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e7b2fb-8608-4235-aea2-b60c1e295270-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:49.138577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.138564    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:49.138646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.138579    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:49.138646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.138593    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:49.139822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.139799    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-podinfo" pod "49e7b2fb-8608-4235-aea2-b60c1e295270" (UID: "49e7b2fb-8608-4235-aea2-b60c1e295270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:43:49.139822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.139804    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-envoy" pod "49e7b2fb-8608-4235-aea2-b60c1e295270" (UID: "49e7b2fb-8608-4235-aea2-b60c1e295270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:43:49.140121 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.140088    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-token" pod "49e7b2fb-8608-4235-aea2-b60c1e295270" (UID: "49e7b2fb-8608-4235-aea2-b60c1e295270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:43:49.140259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.140227    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-kube-api-access-4n28j" pod "49e7b2fb-8608-4235-aea2-b60c1e295270" (UID: "49e7b2fb-8608-4235-aea2-b60c1e295270"). InnerVolumeSpecName "kube-api-access-4n28j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:43:49.239342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.239311    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:49.239342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.239338    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:49.239342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.239348    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e7b2fb-8608-4235-aea2-b60c1e295270-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:49.239565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.239357    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n28j\" (UniqueName: \"kubernetes.io/projected/49e7b2fb-8608-4235-aea2-b60c1e295270-kube-api-access-4n28j\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:43:49.808053 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.808019    4528 generic.go:356] "Generic (PLEG): container finished" podID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerID="bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4" exitCode=0
Nov 02 21:43:49.808250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.808061    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"
Nov 02 21:43:49.808250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.808104    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" event={"ID":"49e7b2fb-8608-4235-aea2-b60c1e295270","Type":"ContainerDied","Data":"bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4"}
Nov 02 21:43:49.808250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.808175    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc" event={"ID":"49e7b2fb-8608-4235-aea2-b60c1e295270","Type":"ContainerDied","Data":"29c9b9195413f284795dffffdfab21b17f29abcfa5d998c701c1f10ff7e431a5"}
Nov 02 21:43:49.808250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.808197    4528 scope.go:122] "RemoveContainer" containerID="bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4"
Nov 02 21:43:49.818407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.818390    4528 scope.go:122] "RemoveContainer" containerID="bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4"
Nov 02 21:43:49.818711 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:43:49.818690    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4\": container with ID starting with bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4 not found: ID does not exist" containerID="bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4"
Nov 02 21:43:49.818755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:49.818722    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4"} err="failed to get container status \"bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4\": rpc error: code = NotFound desc = could not find container \"bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4\": container with ID starting with bec602f0140abe43b4af2047e16024302dd3ac14fc344583e16c052509eadfd4 not found: ID does not exist"
Nov 02 21:43:50.023313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:50.023278    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"]
Nov 02 21:43:50.073980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:50.073846    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bsmqc"]
Nov 02 21:43:50.941375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:50.941342    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" path="/var/lib/kubelet/pods/49e7b2fb-8608-4235-aea2-b60c1e295270/volumes"
Nov 02 21:43:51.731004 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.730972    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"]
Nov 02 21:43:51.731531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.731516    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7" containerName="payload-processing"
Nov 02 21:43:51.731581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.731533    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7" containerName="payload-processing"
Nov 02 21:43:51.731581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.731546    4528 cpu_manager.go:414] "containerMap: removing container" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy"
Nov 02 21:43:51.731581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.731551    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy"
Nov 02 21:43:51.731680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.731648    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6fcdf5-6a58-4dc8-bda7-4ca9bdb77ef7" containerName="payload-processing"
Nov 02 21:43:51.731680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.731656    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e7b2fb-8608-4235-aea2-b60c1e295270" containerName="istio-proxy"
Nov 02 21:43:51.736338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.736321    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:43:51.779484 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.779461    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qffn7\" (UniqueName: \"kubernetes.io/projected/ab3df98c-f72b-4e36-b436-18bbcba2c20a-kube-api-access-qffn7\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc\" (UID: \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:43:51.779623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.779496    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3df98c-f72b-4e36-b436-18bbcba2c20a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc\" (UID: \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:43:51.787135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.787101    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"]
Nov 02 21:43:51.834542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.834511    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a9abc\""
Nov 02 21:43:51.834706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.834624    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a9abc-dockercfg-qqxjn\""
Nov 02 21:43:51.880357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.880332    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qffn7\" (UniqueName: \"kubernetes.io/projected/ab3df98c-f72b-4e36-b436-18bbcba2c20a-kube-api-access-qffn7\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc\" (UID: \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:43:51.880480 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.880368    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3df98c-f72b-4e36-b436-18bbcba2c20a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc\" (UID: \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:43:51.881075 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.881057    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3df98c-f72b-4e36-b436-18bbcba2c20a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc\" (UID: \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:43:51.936881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:51.936854    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qffn7\" (UniqueName: \"kubernetes.io/projected/ab3df98c-f72b-4e36-b436-18bbcba2c20a-kube-api-access-qffn7\") pod \"payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc\" (UID: \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:43:52.049623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.049597    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:43:52.416597 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:43:52.416560    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3df98c_f72b_4e36_b436_18bbcba2c20a.slice/crio-3e10f4064e5f1b5c6f966da45f6915fcbda94f721ba816951a192bee60932e4b WatchSource:0}: Error finding container 3e10f4064e5f1b5c6f966da45f6915fcbda94f721ba816951a192bee60932e4b: Status 404 returned error can't find the container with id 3e10f4064e5f1b5c6f966da45f6915fcbda94f721ba816951a192bee60932e4b
Nov 02 21:43:52.418833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.418809    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"]
Nov 02 21:43:52.515753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.515726    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"]
Nov 02 21:43:52.519801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.519785    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:43:52.571462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.571393    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"]
Nov 02 21:43:52.587292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.587237    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052e3ec-2963-4b40-a013-d2fe85f7f989-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x\" (UID: \"6052e3ec-2963-4b40-a013-d2fe85f7f989\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:43:52.587477 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.587428    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwn8\" (UniqueName: \"kubernetes.io/projected/6052e3ec-2963-4b40-a013-d2fe85f7f989-kube-api-access-9dwn8\") pod \"payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x\" (UID: \"6052e3ec-2963-4b40-a013-d2fe85f7f989\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:43:52.688233 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.688162    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwn8\" (UniqueName: \"kubernetes.io/projected/6052e3ec-2963-4b40-a013-d2fe85f7f989-kube-api-access-9dwn8\") pod \"payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x\" (UID: \"6052e3ec-2963-4b40-a013-d2fe85f7f989\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:43:52.688360 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.688249    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052e3ec-2963-4b40-a013-d2fe85f7f989-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x\" (UID: \"6052e3ec-2963-4b40-a013-d2fe85f7f989\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:43:52.688941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.688912    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052e3ec-2963-4b40-a013-d2fe85f7f989-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x\" (UID: \"6052e3ec-2963-4b40-a013-d2fe85f7f989\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:43:52.745057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.745033    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwn8\" (UniqueName: \"kubernetes.io/projected/6052e3ec-2963-4b40-a013-d2fe85f7f989-kube-api-access-9dwn8\") pod \"payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x\" (UID: \"6052e3ec-2963-4b40-a013-d2fe85f7f989\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:43:52.831998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.831969    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc" event={"ID":"ab3df98c-f72b-4e36-b436-18bbcba2c20a","Type":"ContainerStarted","Data":"918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7"}
Nov 02 21:43:52.832176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.832018    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc" event={"ID":"ab3df98c-f72b-4e36-b436-18bbcba2c20a","Type":"ContainerStarted","Data":"3e10f4064e5f1b5c6f966da45f6915fcbda94f721ba816951a192bee60932e4b"}
Nov 02 21:43:52.832176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.832154    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:43:52.832302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.832228    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:43:52.948406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:52.948309    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc" podStartSLOduration=1.948289907 podStartE2EDuration="1.948289907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:52.946485997 +0000 UTC m=+2120.210022318" watchObservedRunningTime="2026-11-02 21:43:52.948289907 +0000 UTC m=+2120.211826227"
Nov 02 21:43:53.217160 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:43:53.217121    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6052e3ec_2963_4b40_a013_d2fe85f7f989.slice/crio-aa3f56bf9dbcbbc5c1d587681785c4ea8b4498166ad93a7fb52f198d369c9da1 WatchSource:0}: Error finding container aa3f56bf9dbcbbc5c1d587681785c4ea8b4498166ad93a7fb52f198d369c9da1: Status 404 returned error can't find the container with id aa3f56bf9dbcbbc5c1d587681785c4ea8b4498166ad93a7fb52f198d369c9da1
Nov 02 21:43:53.220747 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:53.220724    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"]
Nov 02 21:43:53.838697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:53.838655    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x" event={"ID":"6052e3ec-2963-4b40-a013-d2fe85f7f989","Type":"ContainerStarted","Data":"160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2"}
Nov 02 21:43:53.838895 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:53.838718    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x" event={"ID":"6052e3ec-2963-4b40-a013-d2fe85f7f989","Type":"ContainerStarted","Data":"aa3f56bf9dbcbbc5c1d587681785c4ea8b4498166ad93a7fb52f198d369c9da1"}
Nov 02 21:43:53.838895 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:53.838749    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:43:53.954184 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:43:53.954135    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x" podStartSLOduration=1.954122453 podStartE2EDuration="1.954122453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:53.952820113 +0000 UTC m=+2121.216356431" watchObservedRunningTime="2026-11-02 21:43:53.954122453 +0000 UTC m=+2121.217658761"
Nov 02 21:44:02.115652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:02.115607    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:44:03.841263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:03.841230    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:44:04.849118 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:04.849085    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:44:14.116750 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:44:14.116714    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:44:23.154380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.154348    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"]
Nov 02 21:44:23.158425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.158409    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:23.215430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.210809    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"]
Nov 02 21:44:23.242591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.242556    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26caed4f-a41c-4e8d-8170-36ae88f6b177-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm\" (UID: \"26caed4f-a41c-4e8d-8170-36ae88f6b177\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:23.242776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.242752    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vzq\" (UniqueName: \"kubernetes.io/projected/26caed4f-a41c-4e8d-8170-36ae88f6b177-kube-api-access-d7vzq\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm\" (UID: \"26caed4f-a41c-4e8d-8170-36ae88f6b177\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:23.256619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.256598    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ce2453\""
Nov 02 21:44:23.266210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.266193    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ce2453-dockercfg-mgpnf\""
Nov 02 21:44:23.343941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.343916    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26caed4f-a41c-4e8d-8170-36ae88f6b177-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm\" (UID: \"26caed4f-a41c-4e8d-8170-36ae88f6b177\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:23.344131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.344090    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vzq\" (UniqueName: \"kubernetes.io/projected/26caed4f-a41c-4e8d-8170-36ae88f6b177-kube-api-access-d7vzq\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm\" (UID: \"26caed4f-a41c-4e8d-8170-36ae88f6b177\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:23.344670 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.344652    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26caed4f-a41c-4e8d-8170-36ae88f6b177-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm\" (UID: \"26caed4f-a41c-4e8d-8170-36ae88f6b177\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:23.401280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.401254    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vzq\" (UniqueName: \"kubernetes.io/projected/26caed4f-a41c-4e8d-8170-36ae88f6b177-kube-api-access-d7vzq\") pod \"payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm\" (UID: \"26caed4f-a41c-4e8d-8170-36ae88f6b177\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:23.472082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.472020    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:23.848921 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:44:23.844674    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26caed4f_a41c_4e8d_8170_36ae88f6b177.slice/crio-49b23964e6909a53fc2ab56ddf739e5807c0efed46bd603cbfef3f773cdf8af7 WatchSource:0}: Error finding container 49b23964e6909a53fc2ab56ddf739e5807c0efed46bd603cbfef3f773cdf8af7: Status 404 returned error can't find the container with id 49b23964e6909a53fc2ab56ddf739e5807c0efed46bd603cbfef3f773cdf8af7
Nov 02 21:44:23.848921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.848767    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"]
Nov 02 21:44:23.944450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.944419    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"]
Nov 02 21:44:23.948375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:23.948355    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:24.001898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.001867    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"]
Nov 02 21:44:24.012373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.012341    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm" event={"ID":"26caed4f-a41c-4e8d-8170-36ae88f6b177","Type":"ContainerStarted","Data":"d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154"}
Nov 02 21:44:24.012498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.012403    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm" event={"ID":"26caed4f-a41c-4e8d-8170-36ae88f6b177","Type":"ContainerStarted","Data":"49b23964e6909a53fc2ab56ddf739e5807c0efed46bd603cbfef3f773cdf8af7"}
Nov 02 21:44:24.012498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.012449    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:24.051668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.051640    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77ss2\" (UniqueName: \"kubernetes.io/projected/ef571e2a-0892-492e-a3d8-d57dd8fe5358-kube-api-access-77ss2\") pod \"payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk\" (UID: \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:24.051870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.051852    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef571e2a-0892-492e-a3d8-d57dd8fe5358-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk\" (UID: \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:24.152565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.152488    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef571e2a-0892-492e-a3d8-d57dd8fe5358-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk\" (UID: \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:24.152565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.152545    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77ss2\" (UniqueName: \"kubernetes.io/projected/ef571e2a-0892-492e-a3d8-d57dd8fe5358-kube-api-access-77ss2\") pod \"payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk\" (UID: \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:24.153182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.153160    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef571e2a-0892-492e-a3d8-d57dd8fe5358-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk\" (UID: \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:24.174583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.174535    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm" podStartSLOduration=1.174521226 podStartE2EDuration="1.174521226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:24.174236363 +0000 UTC m=+2151.437772680" watchObservedRunningTime="2026-11-02 21:44:24.174521226 +0000 UTC m=+2151.438057545"
Nov 02 21:44:24.209154 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.209127    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77ss2\" (UniqueName: \"kubernetes.io/projected/ef571e2a-0892-492e-a3d8-d57dd8fe5358-kube-api-access-77ss2\") pod \"payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk\" (UID: \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:24.262336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.262307    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:24.678569 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:44:24.678538    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef571e2a_0892_492e_a3d8_d57dd8fe5358.slice/crio-608db8965f30a6728a0b95afcbe6fbbc61aa62bb02b0f56858be669d1d8043c1 WatchSource:0}: Error finding container 608db8965f30a6728a0b95afcbe6fbbc61aa62bb02b0f56858be669d1d8043c1: Status 404 returned error can't find the container with id 608db8965f30a6728a0b95afcbe6fbbc61aa62bb02b0f56858be669d1d8043c1
Nov 02 21:44:24.681997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:24.681975    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"]
Nov 02 21:44:25.020751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:25.020714    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk" event={"ID":"ef571e2a-0892-492e-a3d8-d57dd8fe5358","Type":"ContainerStarted","Data":"92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e"}
Nov 02 21:44:25.020966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:25.020776    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk" event={"ID":"ef571e2a-0892-492e-a3d8-d57dd8fe5358","Type":"ContainerStarted","Data":"608db8965f30a6728a0b95afcbe6fbbc61aa62bb02b0f56858be669d1d8043c1"}
Nov 02 21:44:25.020966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:25.020805    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:25.137404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:25.137346    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk" podStartSLOduration=2.137328398 podStartE2EDuration="2.137328398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:25.134449691 +0000 UTC m=+2152.397986010" watchObservedRunningTime="2026-11-02 21:44:25.137328398 +0000 UTC m=+2152.400864715"
Nov 02 21:44:30.337040 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.337008    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"]
Nov 02 21:44:30.341037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.341019    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:30.396166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.396135    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"]
Nov 02 21:44:30.432253 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.432225    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b\" (UID: \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:30.432413 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.432395    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbs6j\" (UniqueName: \"kubernetes.io/projected/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-kube-api-access-cbs6j\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b\" (UID: \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:30.442038 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.442013    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-387f88\""
Nov 02 21:44:30.442488 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.442469    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-387f88-dockercfg-55l2v\""
Nov 02 21:44:30.533917 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.533888    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b\" (UID: \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:30.534095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.533993    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbs6j\" (UniqueName: \"kubernetes.io/projected/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-kube-api-access-cbs6j\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b\" (UID: \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:30.534557 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.534538    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b\" (UID: \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:30.590848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.590782    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbs6j\" (UniqueName: \"kubernetes.io/projected/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-kube-api-access-cbs6j\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b\" (UID: \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:30.655169 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:30.655142    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:31.020464 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:44:31.020428    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72f5c1b_6201_4c2d_a8aa_8d17bd7169e2.slice/crio-fd2b469abb3d27040ae72a019eb0a0643d1682b9ca328b8aa71e62f778c68227 WatchSource:0}: Error finding container fd2b469abb3d27040ae72a019eb0a0643d1682b9ca328b8aa71e62f778c68227: Status 404 returned error can't find the container with id fd2b469abb3d27040ae72a019eb0a0643d1682b9ca328b8aa71e62f778c68227
Nov 02 21:44:31.024027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.024007    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"]
Nov 02 21:44:31.057247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.057223    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b" event={"ID":"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2","Type":"ContainerStarted","Data":"fd2b469abb3d27040ae72a019eb0a0643d1682b9ca328b8aa71e62f778c68227"}
Nov 02 21:44:31.110977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.110950    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"]
Nov 02 21:44:31.114903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.114888    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:31.168012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.167978    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"]
Nov 02 21:44:31.242348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.242314    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl64z\" (UniqueName: \"kubernetes.io/projected/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-kube-api-access-nl64z\") pod \"payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q\" (UID: \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:31.242529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.242361    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q\" (UID: \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:31.343277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.343190    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl64z\" (UniqueName: \"kubernetes.io/projected/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-kube-api-access-nl64z\") pod \"payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q\" (UID: \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:31.343277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.343248    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q\" (UID: \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:31.343909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.343887    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q\" (UID: \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:31.399836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.399803    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl64z\" (UniqueName: \"kubernetes.io/projected/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-kube-api-access-nl64z\") pod \"payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q\" (UID: \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:31.429022 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.428991    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:31.782397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.782367    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"]
Nov 02 21:44:31.782601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.782577    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm" podUID="be7bf237-e30a-4102-b487-9f5486b259fa" containerName="payload-processing" containerID="cri-o://8742786dee8113cd6a279824949fd88f18ee8a5b7652f2d9a592489e1ac4027a" gracePeriod=30
Nov 02 21:44:31.820317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.820262    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"]
Nov 02 21:44:31.842863 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.842837    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"]
Nov 02 21:44:31.843048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:31.843028    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j" podUID="bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c" containerName="payload-pre-processing" containerID="cri-o://4685c5f97c736d488dd8b003fbb8cf66ebcda255ebc36534bff98b47942ff120" gracePeriod=30
Nov 02 21:44:31.853593 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:44:31.853562    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91cdb17_b6df_46f8_b1cf_99287b1d68a7.slice/crio-92b53c4048a6ccbdf9924fdde1faf62fd2b77fce792ebb401653c8aee27514a2 WatchSource:0}: Error finding container 92b53c4048a6ccbdf9924fdde1faf62fd2b77fce792ebb401653c8aee27514a2: Status 404 returned error can't find the container with id 92b53c4048a6ccbdf9924fdde1faf62fd2b77fce792ebb401653c8aee27514a2
Nov 02 21:44:32.066823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.066772    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q" event={"ID":"d91cdb17-b6df-46f8-b1cf-99287b1d68a7","Type":"ContainerStarted","Data":"b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64"}
Nov 02 21:44:32.066984 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.066848    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q" event={"ID":"d91cdb17-b6df-46f8-b1cf-99287b1d68a7","Type":"ContainerStarted","Data":"92b53c4048a6ccbdf9924fdde1faf62fd2b77fce792ebb401653c8aee27514a2"}
Nov 02 21:44:32.067447 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.067424    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:32.069539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.069516    4528 generic.go:356] "Generic (PLEG): container finished" podID="be7bf237-e30a-4102-b487-9f5486b259fa" containerID="8742786dee8113cd6a279824949fd88f18ee8a5b7652f2d9a592489e1ac4027a" exitCode=0
Nov 02 21:44:32.069646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.069599    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm" event={"ID":"be7bf237-e30a-4102-b487-9f5486b259fa","Type":"ContainerDied","Data":"8742786dee8113cd6a279824949fd88f18ee8a5b7652f2d9a592489e1ac4027a"}
Nov 02 21:44:32.071507 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.071486    4528 generic.go:356] "Generic (PLEG): container finished" podID="bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c" containerID="4685c5f97c736d488dd8b003fbb8cf66ebcda255ebc36534bff98b47942ff120" exitCode=0
Nov 02 21:44:32.071623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.071573    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j" event={"ID":"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c","Type":"ContainerDied","Data":"4685c5f97c736d488dd8b003fbb8cf66ebcda255ebc36534bff98b47942ff120"}
Nov 02 21:44:32.073273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.073252    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b" event={"ID":"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2","Type":"ContainerStarted","Data":"2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b"}
Nov 02 21:44:32.073524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.073502    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:32.143284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.143256    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:44:32.168795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.168771    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:44:32.198478 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.198437    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q" podStartSLOduration=1.198424631 podStartE2EDuration="1.198424631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:32.196891493 +0000 UTC m=+2159.460427811" watchObservedRunningTime="2026-11-02 21:44:32.198424631 +0000 UTC m=+2159.461960944"
Nov 02 21:44:32.269739 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.269715    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-plugins-config-volume\") pod \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\" (UID: \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\") "
Nov 02 21:44:32.269875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.269797    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7bf237-e30a-4102-b487-9f5486b259fa-kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/be7bf237-e30a-4102-b487-9f5486b259fa-kube-api-access-f87mt\") pod \"be7bf237-e30a-4102-b487-9f5486b259fa\" (UID: \"be7bf237-e30a-4102-b487-9f5486b259fa\") "
Nov 02 21:44:32.269875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.269870    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7bf237-e30a-4102-b487-9f5486b259fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7bf237-e30a-4102-b487-9f5486b259fa-plugins-config-volume\") pod \"be7bf237-e30a-4102-b487-9f5486b259fa\" (UID: \"be7bf237-e30a-4102-b487-9f5486b259fa\") "
Nov 02 21:44:32.270013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.269991    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-kube-api-access-pq2pn\" (UniqueName: \"kubernetes.io/projected/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-kube-api-access-pq2pn\") pod \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\" (UID: \"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c\") "
Nov 02 21:44:32.270238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.270214    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-plugins-config-volume" pod "bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c" (UID: "bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:44:32.270312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.270277    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7bf237-e30a-4102-b487-9f5486b259fa-plugins-config-volume" pod "be7bf237-e30a-4102-b487-9f5486b259fa" (UID: "be7bf237-e30a-4102-b487-9f5486b259fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:44:32.270496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.270481    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:32.270547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.270498    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7bf237-e30a-4102-b487-9f5486b259fa-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:32.272154 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.272129    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-kube-api-access-pq2pn" pod "bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c" (UID: "bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c"). InnerVolumeSpecName "kube-api-access-pq2pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:44:32.272227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.272166    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7bf237-e30a-4102-b487-9f5486b259fa-kube-api-access-f87mt" pod "be7bf237-e30a-4102-b487-9f5486b259fa" (UID: "be7bf237-e30a-4102-b487-9f5486b259fa"). InnerVolumeSpecName "kube-api-access-f87mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:44:32.372082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.372000    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87mt\" (UniqueName: \"kubernetes.io/projected/be7bf237-e30a-4102-b487-9f5486b259fa-kube-api-access-f87mt\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:32.372082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.372027    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq2pn\" (UniqueName: \"kubernetes.io/projected/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c-kube-api-access-pq2pn\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:32.588047 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:32.587976    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b" podStartSLOduration=2.587961511 podStartE2EDuration="2.587961511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:32.586025504 +0000 UTC m=+2159.849561820" watchObservedRunningTime="2026-11-02 21:44:32.587961511 +0000 UTC m=+2159.851497820"
Nov 02 21:44:33.081218 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.081185    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"
Nov 02 21:44:33.081218 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.081203    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm" event={"ID":"be7bf237-e30a-4102-b487-9f5486b259fa","Type":"ContainerDied","Data":"81e86ff2c67e4be6d66bae60ce7fb3e98ac579e01c1afa20681c34bda9f38ecf"}
Nov 02 21:44:33.081466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.081271    4528 scope.go:122] "RemoveContainer" containerID="8742786dee8113cd6a279824949fd88f18ee8a5b7652f2d9a592489e1ac4027a"
Nov 02 21:44:33.089290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.089266    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"
Nov 02 21:44:33.089431 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.089376    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j" event={"ID":"bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c","Type":"ContainerDied","Data":"a717cb920b4381840521e7c8d51646fb13944be242756343c04ed6c16276c31c"}
Nov 02 21:44:33.094122 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.094085    4528 scope.go:122] "RemoveContainer" containerID="4685c5f97c736d488dd8b003fbb8cf66ebcda255ebc36534bff98b47942ff120"
Nov 02 21:44:33.249329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.249206    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"]
Nov 02 21:44:33.256389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.256366    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15253e-69557dfd55-q429j"]
Nov 02 21:44:33.363837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.363805    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"]
Nov 02 21:44:33.368222 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:33.368195    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15253e-7bb545fc84-cfwnm"]
Nov 02 21:44:34.913828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:34.913750    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c" path="/var/lib/kubelet/pods/bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c/volumes"
Nov 02 21:44:34.914308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:34.914122    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7bf237-e30a-4102-b487-9f5486b259fa" path="/var/lib/kubelet/pods/be7bf237-e30a-4102-b487-9f5486b259fa/volumes"
Nov 02 21:44:35.023183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:35.023156    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:44:36.028953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:36.028919    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:44:40.140427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:40.140392    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"]
Nov 02 21:44:40.140810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:40.140626    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy" containerID="cri-o://7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df" gracePeriod=30
Nov 02 21:44:42.782890 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:42.782852    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:44:42.783278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:42.782924    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:44:43.108348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:43.108264    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:44:43.109797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:43.109773    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:44:44.112316 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:44:44.112280    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:44:44.440970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.440880    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"]
Nov 02 21:44:44.441606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.441579    4528 cpu_manager.go:414] "containerMap: removing container" podUID="be7bf237-e30a-4102-b487-9f5486b259fa" containerName="payload-processing"
Nov 02 21:44:44.441606 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.441602    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7bf237-e30a-4102-b487-9f5486b259fa" containerName="payload-processing"
Nov 02 21:44:44.441768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.441632    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c" containerName="payload-pre-processing"
Nov 02 21:44:44.441768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.441637    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c" containerName="payload-pre-processing"
Nov 02 21:44:44.441768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.441721    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdfb90f5-9e93-4f3b-b65f-bffc50b4b85c" containerName="payload-pre-processing"
Nov 02 21:44:44.441768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.441733    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7bf237-e30a-4102-b487-9f5486b259fa" containerName="payload-processing"
Nov 02 21:44:44.446635 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.446618    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:44:44.497515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.497489    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"]
Nov 02 21:44:44.544233 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.544210    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3e3997\""
Nov 02 21:44:44.544367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.544281    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3e3997-dockercfg-gzvq2\""
Nov 02 21:44:44.547759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.547728    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d835758-48be-4b27-b62b-7769662df352-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj\" (UID: \"6d835758-48be-4b27-b62b-7769662df352\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:44:44.547923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.547909    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r6rs\" (UniqueName: \"kubernetes.io/projected/6d835758-48be-4b27-b62b-7769662df352-kube-api-access-6r6rs\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj\" (UID: \"6d835758-48be-4b27-b62b-7769662df352\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:44:44.649433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.649404    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6rs\" (UniqueName: \"kubernetes.io/projected/6d835758-48be-4b27-b62b-7769662df352-kube-api-access-6r6rs\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj\" (UID: \"6d835758-48be-4b27-b62b-7769662df352\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:44:44.649565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.649457    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d835758-48be-4b27-b62b-7769662df352-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj\" (UID: \"6d835758-48be-4b27-b62b-7769662df352\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:44:44.650133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.650102    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d835758-48be-4b27-b62b-7769662df352-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj\" (UID: \"6d835758-48be-4b27-b62b-7769662df352\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:44:44.709843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.709779    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r6rs\" (UniqueName: \"kubernetes.io/projected/6d835758-48be-4b27-b62b-7769662df352-kube-api-access-6r6rs\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj\" (UID: \"6d835758-48be-4b27-b62b-7769662df352\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:44:44.759820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:44.759792    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:44:45.126722 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:44:45.126694    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d835758_48be_4b27_b62b_7769662df352.slice/crio-28bb7b310403040d2d038480e2a07d17fa0d21f08ebee256a900736970d25562 WatchSource:0}: Error finding container 28bb7b310403040d2d038480e2a07d17fa0d21f08ebee256a900736970d25562: Status 404 returned error can't find the container with id 28bb7b310403040d2d038480e2a07d17fa0d21f08ebee256a900736970d25562
Nov 02 21:44:45.130841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.130813    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"]
Nov 02 21:44:45.168284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.168262    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"]
Nov 02 21:44:45.172842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.172826    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:45.180431 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.180409    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj" event={"ID":"6d835758-48be-4b27-b62b-7769662df352","Type":"ContainerStarted","Data":"28bb7b310403040d2d038480e2a07d17fa0d21f08ebee256a900736970d25562"}
Nov 02 21:44:45.231647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.231624    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"]
Nov 02 21:44:45.257959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.257924    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd241a8f-8995-4933-b606-f057639dde3d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz\" (UID: \"fd241a8f-8995-4933-b606-f057639dde3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:45.258068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.258054    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7chcs\" (UniqueName: \"kubernetes.io/projected/fd241a8f-8995-4933-b606-f057639dde3d-kube-api-access-7chcs\") pod \"payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz\" (UID: \"fd241a8f-8995-4933-b606-f057639dde3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:45.359595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.359563    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd241a8f-8995-4933-b606-f057639dde3d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz\" (UID: \"fd241a8f-8995-4933-b606-f057639dde3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:45.359770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.359675    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7chcs\" (UniqueName: \"kubernetes.io/projected/fd241a8f-8995-4933-b606-f057639dde3d-kube-api-access-7chcs\") pod \"payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz\" (UID: \"fd241a8f-8995-4933-b606-f057639dde3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:45.360311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.360288    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd241a8f-8995-4933-b606-f057639dde3d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz\" (UID: \"fd241a8f-8995-4933-b606-f057639dde3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:45.403210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.403187    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:44:45.416611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.416588    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7chcs\" (UniqueName: \"kubernetes.io/projected/fd241a8f-8995-4933-b606-f057639dde3d-kube-api-access-7chcs\") pod \"payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz\" (UID: \"fd241a8f-8995-4933-b606-f057639dde3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:45.460806 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.460786    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-socket\") pod \"f971579a-0d19-4119-9924-df6165cfb992\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") "
Nov 02 21:44:45.460915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.460841    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-certs\") pod \"f971579a-0d19-4119-9924-df6165cfb992\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") "
Nov 02 21:44:45.460915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.460879    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-kube-api-access-8522m\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-kube-api-access-8522m\") pod \"f971579a-0d19-4119-9924-df6165cfb992\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") "
Nov 02 21:44:45.460915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.460899    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-credential-socket\") pod \"f971579a-0d19-4119-9924-df6165cfb992\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") "
Nov 02 21:44:45.461085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.460921    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f971579a-0d19-4119-9924-df6165cfb992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971579a-0d19-4119-9924-df6165cfb992-istio-podinfo\") pod \"f971579a-0d19-4119-9924-df6165cfb992\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") "
Nov 02 21:44:45.461085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.460951    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-envoy\") pod \"f971579a-0d19-4119-9924-df6165cfb992\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") "
Nov 02 21:44:45.461085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461040    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-istio-token\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-istio-token\") pod \"f971579a-0d19-4119-9924-df6165cfb992\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") "
Nov 02 21:44:45.461085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461068    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f971579a-0d19-4119-9924-df6165cfb992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971579a-0d19-4119-9924-df6165cfb992-istiod-ca-cert\") pod \"f971579a-0d19-4119-9924-df6165cfb992\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") "
Nov 02 21:44:45.461334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461085    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-socket" pod "f971579a-0d19-4119-9924-df6165cfb992" (UID: "f971579a-0d19-4119-9924-df6165cfb992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:44:45.461334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461209    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-certs" pod "f971579a-0d19-4119-9924-df6165cfb992" (UID: "f971579a-0d19-4119-9924-df6165cfb992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:44:45.461334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461223    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-credential-socket" pod "f971579a-0d19-4119-9924-df6165cfb992" (UID: "f971579a-0d19-4119-9924-df6165cfb992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:44:45.461334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461249    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-data\") pod \"f971579a-0d19-4119-9924-df6165cfb992\" (UID: \"f971579a-0d19-4119-9924-df6165cfb992\") "
Nov 02 21:44:45.461684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461662    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f971579a-0d19-4119-9924-df6165cfb992-istiod-ca-cert" pod "f971579a-0d19-4119-9924-df6165cfb992" (UID: "f971579a-0d19-4119-9924-df6165cfb992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:44:45.461774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461752    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-data" pod "f971579a-0d19-4119-9924-df6165cfb992" (UID: "f971579a-0d19-4119-9924-df6165cfb992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:44:45.461867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461846    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:45.461867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461866    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:45.462052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461876    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:45.462052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461885    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f971579a-0d19-4119-9924-df6165cfb992-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:45.462052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.461894    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:45.463156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.463095    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-kube-api-access-8522m" pod "f971579a-0d19-4119-9924-df6165cfb992" (UID: "f971579a-0d19-4119-9924-df6165cfb992"). InnerVolumeSpecName "kube-api-access-8522m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:44:45.463238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.463182    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-istio-token" pod "f971579a-0d19-4119-9924-df6165cfb992" (UID: "f971579a-0d19-4119-9924-df6165cfb992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:44:45.463393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.463377    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-envoy" pod "f971579a-0d19-4119-9924-df6165cfb992" (UID: "f971579a-0d19-4119-9924-df6165cfb992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:44:45.463456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.463436    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f971579a-0d19-4119-9924-df6165cfb992-istio-podinfo" pod "f971579a-0d19-4119-9924-df6165cfb992" (UID: "f971579a-0d19-4119-9924-df6165cfb992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:44:45.558232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.558197    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:45.563032 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.563007    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:45.563171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.563037    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8522m\" (UniqueName: \"kubernetes.io/projected/f971579a-0d19-4119-9924-df6165cfb992-kube-api-access-8522m\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:45.563171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.563055    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f971579a-0d19-4119-9924-df6165cfb992-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:45.563171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.563068    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f971579a-0d19-4119-9924-df6165cfb992-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:44:45.937959 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:45.937931    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"]
Nov 02 21:44:46.188903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.188862    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj" event={"ID":"6d835758-48be-4b27-b62b-7769662df352","Type":"ContainerStarted","Data":"8b99132b2cd54c4147e758a0d99f0e5b23009e4d3e00dfb87293c6eddeb1551f"}
Nov 02 21:44:46.189399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.188941    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:44:46.190552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.190522    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz" event={"ID":"fd241a8f-8995-4933-b606-f057639dde3d","Type":"ContainerStarted","Data":"d346c1c473bff8a8f945ce433c9d6eec15c34103c3da6cafbdc4c40fe0d0423d"}
Nov 02 21:44:46.190695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.190559    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz" event={"ID":"fd241a8f-8995-4933-b606-f057639dde3d","Type":"ContainerStarted","Data":"ec5e11a67fe5c42fd8f600ae2f942d3d368d3b64284e3cedde77ae38abb7a00e"}
Nov 02 21:44:46.190695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.190585    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:46.192369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.192346    4528 generic.go:356] "Generic (PLEG): container finished" podID="f971579a-0d19-4119-9924-df6165cfb992" containerID="7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df" exitCode=0
Nov 02 21:44:46.192453 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.192390    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"
Nov 02 21:44:46.192453 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.192422    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" event={"ID":"f971579a-0d19-4119-9924-df6165cfb992","Type":"ContainerDied","Data":"7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df"}
Nov 02 21:44:46.192539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.192465    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq" event={"ID":"f971579a-0d19-4119-9924-df6165cfb992","Type":"ContainerDied","Data":"86d381f62fdc2305eed3fb0174bc89dca604f3d814f8e51c85ae62c5862ef67b"}
Nov 02 21:44:46.192539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.192485    4528 scope.go:122] "RemoveContainer" containerID="7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df"
Nov 02 21:44:46.202544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.202523    4528 scope.go:122] "RemoveContainer" containerID="e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898"
Nov 02 21:44:46.211180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.211159    4528 scope.go:122] "RemoveContainer" containerID="7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df"
Nov 02 21:44:46.211450 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:44:46.211433    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df\": container with ID starting with 7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df not found: ID does not exist" containerID="7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df"
Nov 02 21:44:46.211494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.211456    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df"} err="failed to get container status \"7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df\": rpc error: code = NotFound desc = could not find container \"7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df\": container with ID starting with 7939789987e3e80266d5eb936277ec3c7f7b7bb525134db2f5abb8e3d93bc9df not found: ID does not exist"
Nov 02 21:44:46.211494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.211471    4528 scope.go:122] "RemoveContainer" containerID="e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898"
Nov 02 21:44:46.212518 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:44:46.212492    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898\": container with ID starting with e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898 not found: ID does not exist" containerID="e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898"
Nov 02 21:44:46.212574 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.212524    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898"} err="failed to get container status \"e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898\": rpc error: code = NotFound desc = could not find container \"e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898\": container with ID starting with e9bedd7336b4bde8b26c1d27ec34d3dd7ea0bcffdf832e509b3c946d473bc898 not found: ID does not exist"
Nov 02 21:44:46.305509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.305460    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj" podStartSLOduration=2.305447713 podStartE2EDuration="2.305447713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:46.30270443 +0000 UTC m=+2173.566240748" watchObservedRunningTime="2026-11-02 21:44:46.305447713 +0000 UTC m=+2173.568984030"
Nov 02 21:44:46.417675 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.417636    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"]
Nov 02 21:44:46.424361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.421319    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-15253e-openshift-default-648b584869-zp8cq"]
Nov 02 21:44:46.536423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.536365    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz" podStartSLOduration=1.536349814 podStartE2EDuration="1.536349814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:46.535181294 +0000 UTC m=+2173.798717609" watchObservedRunningTime="2026-11-02 21:44:46.536349814 +0000 UTC m=+2173.799886130"
Nov 02 21:44:46.914552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:46.914518    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f971579a-0d19-4119-9924-df6165cfb992" path="/var/lib/kubelet/pods/f971579a-0d19-4119-9924-df6165cfb992/volumes"
Nov 02 21:44:47.692581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.692546    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"]
Nov 02 21:44:47.694083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.694056    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy"
Nov 02 21:44:47.694264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.694249    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy"
Nov 02 21:44:47.694559 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.694542    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy"
Nov 02 21:44:47.703870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.703596    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:44:47.742870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.742840    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"]
Nov 02 21:44:47.757236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.757210    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aa60bc-dockercfg-7djms\""
Nov 02 21:44:47.789515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.789493    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa60bc\""
Nov 02 21:44:47.797127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.797083    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698428c7-1a90-460d-8afc-b96ad903c217-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc\" (UID: \"698428c7-1a90-460d-8afc-b96ad903c217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:44:47.797220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.797205    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggk5x\" (UniqueName: \"kubernetes.io/projected/698428c7-1a90-460d-8afc-b96ad903c217-kube-api-access-ggk5x\") pod \"payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc\" (UID: \"698428c7-1a90-460d-8afc-b96ad903c217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:44:47.898512 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.898483    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698428c7-1a90-460d-8afc-b96ad903c217-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc\" (UID: \"698428c7-1a90-460d-8afc-b96ad903c217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:44:47.898643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.898556    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggk5x\" (UniqueName: \"kubernetes.io/projected/698428c7-1a90-460d-8afc-b96ad903c217-kube-api-access-ggk5x\") pod \"payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc\" (UID: \"698428c7-1a90-460d-8afc-b96ad903c217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:44:47.899193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.899170    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698428c7-1a90-460d-8afc-b96ad903c217-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc\" (UID: \"698428c7-1a90-460d-8afc-b96ad903c217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:44:47.957323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:47.957270    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggk5x\" (UniqueName: \"kubernetes.io/projected/698428c7-1a90-460d-8afc-b96ad903c217-kube-api-access-ggk5x\") pod \"payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc\" (UID: \"698428c7-1a90-460d-8afc-b96ad903c217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:44:48.024791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:48.024768    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:44:48.392973 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:44:48.392925    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698428c7_1a90_460d_8afc_b96ad903c217.slice/crio-5e23152421654a7d71a7bf19f00e6be2e1e667b6f170855e73d85422967f1471 WatchSource:0}: Error finding container 5e23152421654a7d71a7bf19f00e6be2e1e667b6f170855e73d85422967f1471: Status 404 returned error can't find the container with id 5e23152421654a7d71a7bf19f00e6be2e1e667b6f170855e73d85422967f1471
Nov 02 21:44:48.396356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:48.396328    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"]
Nov 02 21:44:49.214677 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:49.214646    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc" event={"ID":"698428c7-1a90-460d-8afc-b96ad903c217","Type":"ContainerStarted","Data":"758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715"}
Nov 02 21:44:49.215044 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:49.214694    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc" event={"ID":"698428c7-1a90-460d-8afc-b96ad903c217","Type":"ContainerStarted","Data":"5e23152421654a7d71a7bf19f00e6be2e1e667b6f170855e73d85422967f1471"}
Nov 02 21:44:49.215044 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:49.214722    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:44:49.330612 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:49.330564    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc" podStartSLOduration=2.330549971 podStartE2EDuration="2.330549971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:49.328779979 +0000 UTC m=+2176.592316297" watchObservedRunningTime="2026-11-02 21:44:49.330549971 +0000 UTC m=+2176.594086279"
Nov 02 21:44:53.320316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:53.320281    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:44:57.202422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:57.202392    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:44:57.202915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:44:57.202473    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:45:00.165153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.165118    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"]
Nov 02 21:45:00.165700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.165686    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy"
Nov 02 21:45:00.165744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.165700    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy"
Nov 02 21:45:00.165829 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.165820    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f971579a-0d19-4119-9924-df6165cfb992" containerName="istio-proxy"
Nov 02 21:45:00.169324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.169306    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:00.179214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.179194    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"]
Nov 02 21:45:00.188695 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.188678    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"]
Nov 02 21:45:00.188808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.188781    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:00.192910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.192892    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:00.215499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.215475    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"]
Nov 02 21:45:00.219953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.219935    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"]
Nov 02 21:45:00.220080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.220063    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:00.224026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.224008    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:45:00.224249 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.224229    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:00.225195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.225179    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"]
Nov 02 21:45:00.228885 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.228862    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"]
Nov 02 21:45:00.229014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.228962    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:00.232830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.232814    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"]
Nov 02 21:45:00.232914 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.232905    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:00.236420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.236406    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"]
Nov 02 21:45:00.236531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.236516    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:00.240384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.240362    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:00.255973 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.255949    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d0e4-4b05-4e38-8865-d30dd5085528-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4\" (UID: \"bec2d0e4-4b05-4e38-8865-d30dd5085528\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:00.256143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.256069    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd88k\" (UniqueName: \"kubernetes.io/projected/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-kube-api-access-xd88k\") pod \"maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf\" (UID: \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:00.256253 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.256146    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkvp4\" (UniqueName: \"kubernetes.io/projected/938d59c5-d567-4af2-8672-e29bf28a458e-kube-api-access-rkvp4\") pod \"maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v\" (UID: \"938d59c5-d567-4af2-8672-e29bf28a458e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:00.256337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.256271    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938d59c5-d567-4af2-8672-e29bf28a458e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v\" (UID: \"938d59c5-d567-4af2-8672-e29bf28a458e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:00.256337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.256314    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9px7\" (UniqueName: \"kubernetes.io/projected/bec2d0e4-4b05-4e38-8865-d30dd5085528-kube-api-access-s9px7\") pod \"maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4\" (UID: \"bec2d0e4-4b05-4e38-8865-d30dd5085528\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:00.256449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.256398    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf\" (UID: \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:00.262729 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.262689    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:45:00.262729 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.262711    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:45:00.263099 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.263082    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:45:00.266248 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.266229    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"]
Nov 02 21:45:00.275068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.275047    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"]
Nov 02 21:45:00.277459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.277442    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"]
Nov 02 21:45:00.306730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.305023    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"]
Nov 02 21:45:00.319608 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.317311    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"]
Nov 02 21:45:00.321783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.321757    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"]
Nov 02 21:45:00.357891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.357850    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wclhw\" (UniqueName: \"kubernetes.io/projected/2f8e4c51-545b-4043-8bbc-6314c1e91839-kube-api-access-wclhw\") pod \"maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255\" (UID: \"2f8e4c51-545b-4043-8bbc-6314c1e91839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:00.358033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.357918    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2763b03-c0d8-4c1d-8b46-8c6615749031-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md\" (UID: \"a2763b03-c0d8-4c1d-8b46-8c6615749031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:00.358033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.357986    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/987cde20-ab8f-4bce-a527-d57626938569-kube-api-access-p9chl\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4\" (UID: \"987cde20-ab8f-4bce-a527-d57626938569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:00.358149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358047    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd88k\" (UniqueName: \"kubernetes.io/projected/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-kube-api-access-xd88k\") pod \"maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf\" (UID: \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:00.358149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358072    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cde20-ab8f-4bce-a527-d57626938569-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4\" (UID: \"987cde20-ab8f-4bce-a527-d57626938569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:00.358149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358137    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxpwz\" (UniqueName: \"kubernetes.io/projected/453701cb-6a3f-48ae-8cb0-6831a58e06ef-kube-api-access-wxpwz\") pod \"maas-api-key-cleanup-29894265-5scx6\" (UID: \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:00.358273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358164    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkvp4\" (UniqueName: \"kubernetes.io/projected/938d59c5-d567-4af2-8672-e29bf28a458e-kube-api-access-rkvp4\") pod \"maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v\" (UID: \"938d59c5-d567-4af2-8672-e29bf28a458e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:00.358273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358229    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jtx\" (UniqueName: \"kubernetes.io/projected/99712198-d097-4fed-ba4a-76bfa3888fee-kube-api-access-c9jtx\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4\" (UID: \"99712198-d097-4fed-ba4a-76bfa3888fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:00.358354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358290    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453701cb-6a3f-48ae-8cb0-6831a58e06ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-5scx6\" (UID: \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:00.358354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358329    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938d59c5-d567-4af2-8672-e29bf28a458e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v\" (UID: \"938d59c5-d567-4af2-8672-e29bf28a458e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:00.358424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358372    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9px7\" (UniqueName: \"kubernetes.io/projected/bec2d0e4-4b05-4e38-8865-d30dd5085528-kube-api-access-s9px7\") pod \"maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4\" (UID: \"bec2d0e4-4b05-4e38-8865-d30dd5085528\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:00.358465 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358436    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99712198-d097-4fed-ba4a-76bfa3888fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4\" (UID: \"99712198-d097-4fed-ba4a-76bfa3888fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:00.358501 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358480    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf\" (UID: \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:00.358544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358522    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8e4c51-545b-4043-8bbc-6314c1e91839-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255\" (UID: \"2f8e4c51-545b-4043-8bbc-6314c1e91839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:00.358597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358565    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlhh\" (UniqueName: \"kubernetes.io/projected/f3b328bc-d9e2-4821-8e0e-3366a1d84799-kube-api-access-khlhh\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk\" (UID: \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:00.358657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358593    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p9wj\" (UniqueName: \"kubernetes.io/projected/a2763b03-c0d8-4c1d-8b46-8c6615749031-kube-api-access-9p9wj\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md\" (UID: \"a2763b03-c0d8-4c1d-8b46-8c6615749031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:00.358657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358621    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d0e4-4b05-4e38-8865-d30dd5085528-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4\" (UID: \"bec2d0e4-4b05-4e38-8865-d30dd5085528\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:00.358657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.358647    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b328bc-d9e2-4821-8e0e-3366a1d84799-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk\" (UID: \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:00.360542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.360512    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938d59c5-d567-4af2-8672-e29bf28a458e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v\" (UID: \"938d59c5-d567-4af2-8672-e29bf28a458e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:00.361319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.361296    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"]
Nov 02 21:45:00.361421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.361372    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf\" (UID: \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:00.361964 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.361945    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d0e4-4b05-4e38-8865-d30dd5085528-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4\" (UID: \"bec2d0e4-4b05-4e38-8865-d30dd5085528\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:00.362069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.361984    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"]
Nov 02 21:45:00.366056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.366031    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"]
Nov 02 21:45:00.416319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.416250    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkvp4\" (UniqueName: \"kubernetes.io/projected/938d59c5-d567-4af2-8672-e29bf28a458e-kube-api-access-rkvp4\") pod \"maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v\" (UID: \"938d59c5-d567-4af2-8672-e29bf28a458e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:00.416319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.416273    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd88k\" (UniqueName: \"kubernetes.io/projected/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-kube-api-access-xd88k\") pod \"maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf\" (UID: \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:00.416518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.416485    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9px7\" (UniqueName: \"kubernetes.io/projected/bec2d0e4-4b05-4e38-8865-d30dd5085528-kube-api-access-s9px7\") pod \"maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4\" (UID: \"bec2d0e4-4b05-4e38-8865-d30dd5085528\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:00.459723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.459694    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/987cde20-ab8f-4bce-a527-d57626938569-kube-api-access-p9chl\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4\" (UID: \"987cde20-ab8f-4bce-a527-d57626938569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:00.459823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.459738    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cde20-ab8f-4bce-a527-d57626938569-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4\" (UID: \"987cde20-ab8f-4bce-a527-d57626938569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:00.459823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.459777    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxpwz\" (UniqueName: \"kubernetes.io/projected/453701cb-6a3f-48ae-8cb0-6831a58e06ef-kube-api-access-wxpwz\") pod \"maas-api-key-cleanup-29894265-5scx6\" (UID: \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:00.459940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.459835    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jtx\" (UniqueName: \"kubernetes.io/projected/99712198-d097-4fed-ba4a-76bfa3888fee-kube-api-access-c9jtx\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4\" (UID: \"99712198-d097-4fed-ba4a-76bfa3888fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:00.459940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.459884    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453701cb-6a3f-48ae-8cb0-6831a58e06ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-5scx6\" (UID: \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:00.460050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.459962    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99712198-d097-4fed-ba4a-76bfa3888fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4\" (UID: \"99712198-d097-4fed-ba4a-76bfa3888fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:00.460050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8e4c51-545b-4043-8bbc-6314c1e91839-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255\" (UID: \"2f8e4c51-545b-4043-8bbc-6314c1e91839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:00.460183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460056    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlhh\" (UniqueName: \"kubernetes.io/projected/f3b328bc-d9e2-4821-8e0e-3366a1d84799-kube-api-access-khlhh\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk\" (UID: \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:00.460183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p9wj\" (UniqueName: \"kubernetes.io/projected/a2763b03-c0d8-4c1d-8b46-8c6615749031-kube-api-access-9p9wj\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md\" (UID: \"a2763b03-c0d8-4c1d-8b46-8c6615749031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:00.460183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460143    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b328bc-d9e2-4821-8e0e-3366a1d84799-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk\" (UID: \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:00.460352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460191    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wclhw\" (UniqueName: \"kubernetes.io/projected/2f8e4c51-545b-4043-8bbc-6314c1e91839-kube-api-access-wclhw\") pod \"maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255\" (UID: \"2f8e4c51-545b-4043-8bbc-6314c1e91839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:00.460352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460233    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2763b03-c0d8-4c1d-8b46-8c6615749031-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md\" (UID: \"a2763b03-c0d8-4c1d-8b46-8c6615749031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:00.460626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460605    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cde20-ab8f-4bce-a527-d57626938569-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4\" (UID: \"987cde20-ab8f-4bce-a527-d57626938569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:00.460719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460696    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453701cb-6a3f-48ae-8cb0-6831a58e06ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-5scx6\" (UID: \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:00.460850 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460831    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99712198-d097-4fed-ba4a-76bfa3888fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4\" (UID: \"99712198-d097-4fed-ba4a-76bfa3888fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:00.460889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460831    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8e4c51-545b-4043-8bbc-6314c1e91839-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255\" (UID: \"2f8e4c51-545b-4043-8bbc-6314c1e91839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:00.460922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460908    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b328bc-d9e2-4821-8e0e-3366a1d84799-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk\" (UID: \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:00.460964 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.460908    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2763b03-c0d8-4c1d-8b46-8c6615749031-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md\" (UID: \"a2763b03-c0d8-4c1d-8b46-8c6615749031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:00.496454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.496431    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:00.503697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.503676    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:00.511687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.511662    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:00.518029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.518003    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlhh\" (UniqueName: \"kubernetes.io/projected/f3b328bc-d9e2-4821-8e0e-3366a1d84799-kube-api-access-khlhh\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk\" (UID: \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:00.518858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.518835    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxpwz\" (UniqueName: \"kubernetes.io/projected/453701cb-6a3f-48ae-8cb0-6831a58e06ef-kube-api-access-wxpwz\") pod \"maas-api-key-cleanup-29894265-5scx6\" (UID: \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:00.519147 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.518953    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/987cde20-ab8f-4bce-a527-d57626938569-kube-api-access-p9chl\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4\" (UID: \"987cde20-ab8f-4bce-a527-d57626938569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:00.519593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.519568    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p9wj\" (UniqueName: \"kubernetes.io/projected/a2763b03-c0d8-4c1d-8b46-8c6615749031-kube-api-access-9p9wj\") pod \"maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md\" (UID: \"a2763b03-c0d8-4c1d-8b46-8c6615749031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:00.520169 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.520147    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wclhw\" (UniqueName: \"kubernetes.io/projected/2f8e4c51-545b-4043-8bbc-6314c1e91839-kube-api-access-wclhw\") pod \"maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255\" (UID: \"2f8e4c51-545b-4043-8bbc-6314c1e91839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:00.520827 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.520801    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jtx\" (UniqueName: \"kubernetes.io/projected/99712198-d097-4fed-ba4a-76bfa3888fee-kube-api-access-c9jtx\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4\" (UID: \"99712198-d097-4fed-ba4a-76bfa3888fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:00.538386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.538170    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:00.548168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.547953    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:00.554852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.554671    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:00.569625 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.561985    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:00.570934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.570740    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:00.580805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.580578    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:00.883916 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:00.883885    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad43ab3f_245e_4af2_9b9e_0de944ccc0ce.slice/crio-67872d5d7bca527b56855d6d9b0b5752a99591a3640b33a062d7544a307642d9 WatchSource:0}: Error finding container 67872d5d7bca527b56855d6d9b0b5752a99591a3640b33a062d7544a307642d9: Status 404 returned error can't find the container with id 67872d5d7bca527b56855d6d9b0b5752a99591a3640b33a062d7544a307642d9
Nov 02 21:45:00.891825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:00.887143    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"]
Nov 02 21:45:01.043513 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:01.043446    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453701cb_6a3f_48ae_8cb0_6831a58e06ef.slice/crio-a086cfc37e836cfc45cd43d841d2d3c852442180d1cd8c8ec8e437c752778947 WatchSource:0}: Error finding container a086cfc37e836cfc45cd43d841d2d3c852442180d1cd8c8ec8e437c752778947: Status 404 returned error can't find the container with id a086cfc37e836cfc45cd43d841d2d3c852442180d1cd8c8ec8e437c752778947
Nov 02 21:45:01.044149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.044097    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"]
Nov 02 21:45:01.049848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.049822    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"]
Nov 02 21:45:01.052600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.052581    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"]
Nov 02 21:45:01.056574 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:01.054244    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99712198_d097_4fed_ba4a_76bfa3888fee.slice/crio-87f1d4019e3ca2c2f4d66777607a4558d8a1dac36b9c3adddd6305c135b7dc8f WatchSource:0}: Error finding container 87f1d4019e3ca2c2f4d66777607a4558d8a1dac36b9c3adddd6305c135b7dc8f: Status 404 returned error can't find the container with id 87f1d4019e3ca2c2f4d66777607a4558d8a1dac36b9c3adddd6305c135b7dc8f
Nov 02 21:45:01.078587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.076516    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"]
Nov 02 21:45:01.093338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.093315    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"]
Nov 02 21:45:01.099587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.096898    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"]
Nov 02 21:45:01.201364 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:01.201322    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987cde20_ab8f_4bce_a527_d57626938569.slice/crio-9a69ee0d3be38818c50f9dd913dfb665e3f5cf801bae9ba73373b96c28051be9 WatchSource:0}: Error finding container 9a69ee0d3be38818c50f9dd913dfb665e3f5cf801bae9ba73373b96c28051be9: Status 404 returned error can't find the container with id 9a69ee0d3be38818c50f9dd913dfb665e3f5cf801bae9ba73373b96c28051be9
Nov 02 21:45:01.206294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.206271    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"]
Nov 02 21:45:01.263408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.263386    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"]
Nov 02 21:45:01.288791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.288763    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" event={"ID":"99712198-d097-4fed-ba4a-76bfa3888fee","Type":"ContainerStarted","Data":"87f1d4019e3ca2c2f4d66777607a4558d8a1dac36b9c3adddd6305c135b7dc8f"}
Nov 02 21:45:01.289956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.289931    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" event={"ID":"bec2d0e4-4b05-4e38-8865-d30dd5085528","Type":"ContainerStarted","Data":"5898e007aad9d6bc95a74fd832be34d5669550c4981655fe72f366ca644b8713"}
Nov 02 21:45:01.291086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.291065    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf" event={"ID":"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce","Type":"ContainerStarted","Data":"67872d5d7bca527b56855d6d9b0b5752a99591a3640b33a062d7544a307642d9"}
Nov 02 21:45:01.292384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.292362    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" event={"ID":"987cde20-ab8f-4bce-a527-d57626938569","Type":"ContainerStarted","Data":"9a69ee0d3be38818c50f9dd913dfb665e3f5cf801bae9ba73373b96c28051be9"}
Nov 02 21:45:01.293545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.293524    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" event={"ID":"a2763b03-c0d8-4c1d-8b46-8c6615749031","Type":"ContainerStarted","Data":"5d0429b4dbb1accf576c0856bf26af3a1e88655fdbc2349042eb870c4a63fe9e"}
Nov 02 21:45:01.294654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.294634    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" event={"ID":"453701cb-6a3f-48ae-8cb0-6831a58e06ef","Type":"ContainerStarted","Data":"a086cfc37e836cfc45cd43d841d2d3c852442180d1cd8c8ec8e437c752778947"}
Nov 02 21:45:01.295796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.295768    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v" event={"ID":"938d59c5-d567-4af2-8672-e29bf28a458e","Type":"ContainerStarted","Data":"fb340b4edae81916252e1f147b70a214f6dc33aa7201c7736c49019b7482ca62"}
Nov 02 21:45:01.296865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:01.296843    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" event={"ID":"f3b328bc-d9e2-4821-8e0e-3366a1d84799","Type":"ContainerStarted","Data":"33f77e58e8154aadcb2e232c80a8d15c956ed525aacf926d5f5b996ed52c586a"}
Nov 02 21:45:01.309745 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:01.309718    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8e4c51_545b_4043_8bbc_6314c1e91839.slice/crio-5b7171e4dd405880a03050c878fd95570a089f47447e9d21c2bde26a5431a1bd WatchSource:0}: Error finding container 5b7171e4dd405880a03050c878fd95570a089f47447e9d21c2bde26a5431a1bd: Status 404 returned error can't find the container with id 5b7171e4dd405880a03050c878fd95570a089f47447e9d21c2bde26a5431a1bd
Nov 02 21:45:02.306756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.306721    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" event={"ID":"987cde20-ab8f-4bce-a527-d57626938569","Type":"ContainerStarted","Data":"e62c57e7bc92e23244d027ab043a326264d58e7b9733315eca4f3ed65dd2259c"}
Nov 02 21:45:02.308931 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.308905    4528 generic.go:356] "Generic (PLEG): container finished" podID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerID="6662b9eb29142b0b856f7026f8dd100a11d5231d8c900366673d56ee292afe93" exitCode=60
Nov 02 21:45:02.309357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.309338    4528 scope.go:122] "RemoveContainer" containerID="6662b9eb29142b0b856f7026f8dd100a11d5231d8c900366673d56ee292afe93"
Nov 02 21:45:02.309753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.309726    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" event={"ID":"a2763b03-c0d8-4c1d-8b46-8c6615749031","Type":"ContainerDied","Data":"6662b9eb29142b0b856f7026f8dd100a11d5231d8c900366673d56ee292afe93"}
Nov 02 21:45:02.311785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.311759    4528 generic.go:356] "Generic (PLEG): container finished" podID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerID="a31e148f554b8699492926254d35a98259de43a2d993993a8fa89e3ef92b48ac" exitCode=60
Nov 02 21:45:02.311892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.311844    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" event={"ID":"453701cb-6a3f-48ae-8cb0-6831a58e06ef","Type":"ContainerDied","Data":"a31e148f554b8699492926254d35a98259de43a2d993993a8fa89e3ef92b48ac"}
Nov 02 21:45:02.312036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.312018    4528 scope.go:122] "RemoveContainer" containerID="a31e148f554b8699492926254d35a98259de43a2d993993a8fa89e3ef92b48ac"
Nov 02 21:45:02.313632 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.313584    4528 generic.go:356] "Generic (PLEG): container finished" podID="938d59c5-d567-4af2-8672-e29bf28a458e" containerID="57c1b83c14e4bc079c231ff507ae640f88ac93d9fe6100c99b707f9d089ea589" exitCode=60
Nov 02 21:45:02.313706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.313666    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v" event={"ID":"938d59c5-d567-4af2-8672-e29bf28a458e","Type":"ContainerDied","Data":"57c1b83c14e4bc079c231ff507ae640f88ac93d9fe6100c99b707f9d089ea589"}
Nov 02 21:45:02.313826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.313811    4528 scope.go:122] "RemoveContainer" containerID="57c1b83c14e4bc079c231ff507ae640f88ac93d9fe6100c99b707f9d089ea589"
Nov 02 21:45:02.315756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.315737    4528 generic.go:356] "Generic (PLEG): container finished" podID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerID="22cffabb583a32f7f828309b0146111efd3313439585d546ffdfbced5d3196a4" exitCode=60
Nov 02 21:45:02.315880 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.315865    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" event={"ID":"f3b328bc-d9e2-4821-8e0e-3366a1d84799","Type":"ContainerDied","Data":"22cffabb583a32f7f828309b0146111efd3313439585d546ffdfbced5d3196a4"}
Nov 02 21:45:02.316056 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.316044    4528 scope.go:122] "RemoveContainer" containerID="22cffabb583a32f7f828309b0146111efd3313439585d546ffdfbced5d3196a4"
Nov 02 21:45:02.317588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.317568    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" event={"ID":"99712198-d097-4fed-ba4a-76bfa3888fee","Type":"ContainerStarted","Data":"444a3e295ba428050ef7f28446331526dfbdc8509bac24f6ec4e52fdc02078d5"}
Nov 02 21:45:02.317745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.317729    4528 scope.go:122] "RemoveContainer" containerID="444a3e295ba428050ef7f28446331526dfbdc8509bac24f6ec4e52fdc02078d5"
Nov 02 21:45:02.319079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.319060    4528 generic.go:356] "Generic (PLEG): container finished" podID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerID="19dffbb95d8a9ba3caa207fc5dd1cdcedc5f1f90d26d39b630366e6cc26f65f1" exitCode=60
Nov 02 21:45:02.319209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.319147    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" event={"ID":"bec2d0e4-4b05-4e38-8865-d30dd5085528","Type":"ContainerDied","Data":"19dffbb95d8a9ba3caa207fc5dd1cdcedc5f1f90d26d39b630366e6cc26f65f1"}
Nov 02 21:45:02.319344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.319329    4528 scope.go:122] "RemoveContainer" containerID="19dffbb95d8a9ba3caa207fc5dd1cdcedc5f1f90d26d39b630366e6cc26f65f1"
Nov 02 21:45:02.320793 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.320771    4528 generic.go:356] "Generic (PLEG): container finished" podID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" containerID="6261682d8592861fe150960e5ac0ecf7aff9cb128a5e796e7c8d9681a46a9108" exitCode=60
Nov 02 21:45:02.320930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.320898    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf" event={"ID":"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce","Type":"ContainerDied","Data":"6261682d8592861fe150960e5ac0ecf7aff9cb128a5e796e7c8d9681a46a9108"}
Nov 02 21:45:02.321063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.321045    4528 scope.go:122] "RemoveContainer" containerID="6261682d8592861fe150960e5ac0ecf7aff9cb128a5e796e7c8d9681a46a9108"
Nov 02 21:45:02.329855 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.329837    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" event={"ID":"2f8e4c51-545b-4043-8bbc-6314c1e91839","Type":"ContainerStarted","Data":"0524738e8295deaa70efd13905fd70e806a8df00d82e7a41cb3bbd438a0eb877"}
Nov 02 21:45:02.329936 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.329869    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" event={"ID":"2f8e4c51-545b-4043-8bbc-6314c1e91839","Type":"ContainerStarted","Data":"5b7171e4dd405880a03050c878fd95570a089f47447e9d21c2bde26a5431a1bd"}
Nov 02 21:45:02.419537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:02.419489    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" podStartSLOduration=2.419476246 podStartE2EDuration="2.419476246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:02.419088823 +0000 UTC m=+2189.682625140" watchObservedRunningTime="2026-11-02 21:45:02.419476246 +0000 UTC m=+2189.683012562"
Nov 02 21:45:03.095334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.095284    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" podStartSLOduration=3.095270631 podStartE2EDuration="3.095270631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:03.092344197 +0000 UTC m=+2190.355880504" watchObservedRunningTime="2026-11-02 21:45:03.095270631 +0000 UTC m=+2190.358806994"
Nov 02 21:45:03.342174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.342103    4528 generic.go:356] "Generic (PLEG): container finished" podID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerID="0524738e8295deaa70efd13905fd70e806a8df00d82e7a41cb3bbd438a0eb877" exitCode=60
Nov 02 21:45:03.342174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.342156    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" event={"ID":"2f8e4c51-545b-4043-8bbc-6314c1e91839","Type":"ContainerDied","Data":"0524738e8295deaa70efd13905fd70e806a8df00d82e7a41cb3bbd438a0eb877"}
Nov 02 21:45:03.342696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.342472    4528 scope.go:122] "RemoveContainer" containerID="0524738e8295deaa70efd13905fd70e806a8df00d82e7a41cb3bbd438a0eb877"
Nov 02 21:45:03.343870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.343846    4528 generic.go:356] "Generic (PLEG): container finished" podID="987cde20-ab8f-4bce-a527-d57626938569" containerID="e62c57e7bc92e23244d027ab043a326264d58e7b9733315eca4f3ed65dd2259c" exitCode=60
Nov 02 21:45:03.343991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.343927    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" event={"ID":"987cde20-ab8f-4bce-a527-d57626938569","Type":"ContainerDied","Data":"e62c57e7bc92e23244d027ab043a326264d58e7b9733315eca4f3ed65dd2259c"}
Nov 02 21:45:03.344150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.344130    4528 scope.go:122] "RemoveContainer" containerID="e62c57e7bc92e23244d027ab043a326264d58e7b9733315eca4f3ed65dd2259c"
Nov 02 21:45:03.346543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.346520    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" event={"ID":"a2763b03-c0d8-4c1d-8b46-8c6615749031","Type":"ContainerStarted","Data":"05d0b04e2128e432f64f89e2ade53027e53bbab708c0eb276d71c84e1470db6f"}
Nov 02 21:45:03.348458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.348434    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" event={"ID":"453701cb-6a3f-48ae-8cb0-6831a58e06ef","Type":"ContainerStarted","Data":"626f79c1f95e184769beca6b9b4a20e4ddf59c823509c14c08058486048455d8"}
Nov 02 21:45:03.350295 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.350272    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v" event={"ID":"938d59c5-d567-4af2-8672-e29bf28a458e","Type":"ContainerStarted","Data":"4eae84897c6d37d26b956919e60a3d7660a3b79c6cfd0380662bc62ac2aac158"}
Nov 02 21:45:03.352012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.351990    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" event={"ID":"f3b328bc-d9e2-4821-8e0e-3366a1d84799","Type":"ContainerStarted","Data":"7777f07f3b1e9aad0c330b6887bb663d6d0a655544af66967056eb8a214ae1ee"}
Nov 02 21:45:03.354072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.354042    4528 generic.go:356] "Generic (PLEG): container finished" podID="99712198-d097-4fed-ba4a-76bfa3888fee" containerID="444a3e295ba428050ef7f28446331526dfbdc8509bac24f6ec4e52fdc02078d5" exitCode=60
Nov 02 21:45:03.354251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.354149    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" event={"ID":"99712198-d097-4fed-ba4a-76bfa3888fee","Type":"ContainerDied","Data":"444a3e295ba428050ef7f28446331526dfbdc8509bac24f6ec4e52fdc02078d5"}
Nov 02 21:45:03.354251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.354193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" event={"ID":"99712198-d097-4fed-ba4a-76bfa3888fee","Type":"ContainerStarted","Data":"8a24775af519ede616c2aac5ea73405cba0d71a50f922df5d0d1e503d693e234"}
Nov 02 21:45:03.355967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.355949    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" event={"ID":"bec2d0e4-4b05-4e38-8865-d30dd5085528","Type":"ContainerStarted","Data":"26de1ef5926c80a65727e8ff00e305bc58938885a11d5ed4cf1cfd3011d289f5"}
Nov 02 21:45:03.357803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.357784    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf" event={"ID":"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce","Type":"ContainerStarted","Data":"dd47cdba4f5c168c704e0f758eeccae87a80a9dc8f5a894b990b42f79796d720"}
Nov 02 21:45:03.571592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.571541    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" podStartSLOduration=3.5715283639999997 podStartE2EDuration="3.571528364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:03.56897837 +0000 UTC m=+2190.832514686" watchObservedRunningTime="2026-11-02 21:45:03.571528364 +0000 UTC m=+2190.835064681"
Nov 02 21:45:03.793174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.793105    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" podStartSLOduration=3.793088613 podStartE2EDuration="3.793088613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:03.793070718 +0000 UTC m=+2191.056607035" watchObservedRunningTime="2026-11-02 21:45:03.793088613 +0000 UTC m=+2191.056624930"
Nov 02 21:45:03.908718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:03.908666    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" podStartSLOduration=3.9086518139999997 podStartE2EDuration="3.908651814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:03.906787313 +0000 UTC m=+2191.170323630" watchObservedRunningTime="2026-11-02 21:45:03.908651814 +0000 UTC m=+2191.172188161"
Nov 02 21:45:04.020660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.020597    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v" podStartSLOduration=4.020577529 podStartE2EDuration="4.020577529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:04.019296964 +0000 UTC m=+2191.282833302" watchObservedRunningTime="2026-11-02 21:45:04.020577529 +0000 UTC m=+2191.284113846"
Nov 02 21:45:04.134948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.134860    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" podStartSLOduration=4.134848212 podStartE2EDuration="4.134848212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:04.132851198 +0000 UTC m=+2191.396387526" watchObservedRunningTime="2026-11-02 21:45:04.134848212 +0000 UTC m=+2191.398384529"
Nov 02 21:45:04.246414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.246359    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" podStartSLOduration=4.246341985 podStartE2EDuration="4.246341985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:04.245840314 +0000 UTC m=+2191.509376652" watchObservedRunningTime="2026-11-02 21:45:04.246341985 +0000 UTC m=+2191.509878303"
Nov 02 21:45:04.361483 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.361431    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf" podStartSLOduration=4.361416778 podStartE2EDuration="4.361416778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:04.358819884 +0000 UTC m=+2191.622356201" watchObservedRunningTime="2026-11-02 21:45:04.361416778 +0000 UTC m=+2191.624953093"
Nov 02 21:45:04.368699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.368667    4528 generic.go:356] "Generic (PLEG): container finished" podID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerID="05d0b04e2128e432f64f89e2ade53027e53bbab708c0eb276d71c84e1470db6f" exitCode=60
Nov 02 21:45:04.368877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.368745    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" event={"ID":"a2763b03-c0d8-4c1d-8b46-8c6615749031","Type":"ContainerDied","Data":"05d0b04e2128e432f64f89e2ade53027e53bbab708c0eb276d71c84e1470db6f"}
Nov 02 21:45:04.368877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.368801    4528 scope.go:122] "RemoveContainer" containerID="6662b9eb29142b0b856f7026f8dd100a11d5231d8c900366673d56ee292afe93"
Nov 02 21:45:04.369048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.369029    4528 scope.go:122] "RemoveContainer" containerID="05d0b04e2128e432f64f89e2ade53027e53bbab708c0eb276d71c84e1470db6f"
Nov 02 21:45:04.369410 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:04.369378    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md_odh-ai-gateway-infra(a2763b03-c0d8-4c1d-8b46-8c6615749031)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031"
Nov 02 21:45:04.370571 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.370547    4528 generic.go:356] "Generic (PLEG): container finished" podID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerID="626f79c1f95e184769beca6b9b4a20e4ddf59c823509c14c08058486048455d8" exitCode=60
Nov 02 21:45:04.370668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.370613    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" event={"ID":"453701cb-6a3f-48ae-8cb0-6831a58e06ef","Type":"ContainerDied","Data":"626f79c1f95e184769beca6b9b4a20e4ddf59c823509c14c08058486048455d8"}
Nov 02 21:45:04.370889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.370871    4528 scope.go:122] "RemoveContainer" containerID="626f79c1f95e184769beca6b9b4a20e4ddf59c823509c14c08058486048455d8"
Nov 02 21:45:04.371188 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:04.371169    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-5scx6_odh-ai-gateway-infra(453701cb-6a3f-48ae-8cb0-6831a58e06ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef"
Nov 02 21:45:04.372359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.372342    4528 generic.go:356] "Generic (PLEG): container finished" podID="938d59c5-d567-4af2-8672-e29bf28a458e" containerID="4eae84897c6d37d26b956919e60a3d7660a3b79c6cfd0380662bc62ac2aac158" exitCode=60
Nov 02 21:45:04.372439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.372421    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v" event={"ID":"938d59c5-d567-4af2-8672-e29bf28a458e","Type":"ContainerDied","Data":"4eae84897c6d37d26b956919e60a3d7660a3b79c6cfd0380662bc62ac2aac158"}
Nov 02 21:45:04.372675 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.372597    4528 scope.go:122] "RemoveContainer" containerID="4eae84897c6d37d26b956919e60a3d7660a3b79c6cfd0380662bc62ac2aac158"
Nov 02 21:45:04.372855 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:04.372835    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v_odh-ai-gateway-infra(938d59c5-d567-4af2-8672-e29bf28a458e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v" podUID="938d59c5-d567-4af2-8672-e29bf28a458e"
Nov 02 21:45:04.374243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.374225    4528 generic.go:356] "Generic (PLEG): container finished" podID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerID="7777f07f3b1e9aad0c330b6887bb663d6d0a655544af66967056eb8a214ae1ee" exitCode=60
Nov 02 21:45:04.374338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.374252    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" event={"ID":"f3b328bc-d9e2-4821-8e0e-3366a1d84799","Type":"ContainerDied","Data":"7777f07f3b1e9aad0c330b6887bb663d6d0a655544af66967056eb8a214ae1ee"}
Nov 02 21:45:04.374630 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.374611    4528 scope.go:122] "RemoveContainer" containerID="7777f07f3b1e9aad0c330b6887bb663d6d0a655544af66967056eb8a214ae1ee"
Nov 02 21:45:04.374899 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:04.374875    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk_odh-ai-gateway-infra(f3b328bc-d9e2-4821-8e0e-3366a1d84799)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799"
Nov 02 21:45:04.376280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.376259    4528 generic.go:356] "Generic (PLEG): container finished" podID="99712198-d097-4fed-ba4a-76bfa3888fee" containerID="8a24775af519ede616c2aac5ea73405cba0d71a50f922df5d0d1e503d693e234" exitCode=60
Nov 02 21:45:04.376372 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.376337    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" event={"ID":"99712198-d097-4fed-ba4a-76bfa3888fee","Type":"ContainerDied","Data":"8a24775af519ede616c2aac5ea73405cba0d71a50f922df5d0d1e503d693e234"}
Nov 02 21:45:04.376541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.376526    4528 scope.go:122] "RemoveContainer" containerID="8a24775af519ede616c2aac5ea73405cba0d71a50f922df5d0d1e503d693e234"
Nov 02 21:45:04.376757 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:04.376730    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4_odh-ai-gateway-infra(99712198-d097-4fed-ba4a-76bfa3888fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" podUID="99712198-d097-4fed-ba4a-76bfa3888fee"
Nov 02 21:45:04.378186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.378164    4528 generic.go:356] "Generic (PLEG): container finished" podID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerID="26de1ef5926c80a65727e8ff00e305bc58938885a11d5ed4cf1cfd3011d289f5" exitCode=60
Nov 02 21:45:04.378274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.378228    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" event={"ID":"bec2d0e4-4b05-4e38-8865-d30dd5085528","Type":"ContainerDied","Data":"26de1ef5926c80a65727e8ff00e305bc58938885a11d5ed4cf1cfd3011d289f5"}
Nov 02 21:45:04.378451 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.378435    4528 scope.go:122] "RemoveContainer" containerID="26de1ef5926c80a65727e8ff00e305bc58938885a11d5ed4cf1cfd3011d289f5"
Nov 02 21:45:04.378657 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:04.378633    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4_odh-ai-gateway-infra(bec2d0e4-4b05-4e38-8865-d30dd5085528)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528"
Nov 02 21:45:04.380031 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.380012    4528 generic.go:356] "Generic (PLEG): container finished" podID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" containerID="dd47cdba4f5c168c704e0f758eeccae87a80a9dc8f5a894b990b42f79796d720" exitCode=60
Nov 02 21:45:04.380151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.380084    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf" event={"ID":"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce","Type":"ContainerDied","Data":"dd47cdba4f5c168c704e0f758eeccae87a80a9dc8f5a894b990b42f79796d720"}
Nov 02 21:45:04.380286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.380271    4528 scope.go:122] "RemoveContainer" containerID="dd47cdba4f5c168c704e0f758eeccae87a80a9dc8f5a894b990b42f79796d720"
Nov 02 21:45:04.380486 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:04.380465    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf_odh-ai-gateway-infra(ad43ab3f-245e-4af2-9b9e-0de944ccc0ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf" podUID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce"
Nov 02 21:45:04.381951 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.381928    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" event={"ID":"2f8e4c51-545b-4043-8bbc-6314c1e91839","Type":"ContainerStarted","Data":"143cdcb3e0bd6e80d9cb0ee59ef25724a6fa612ac09c6fccb95e41768cff3ea5"}
Nov 02 21:45:04.383367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.383350    4528 scope.go:122] "RemoveContainer" containerID="a31e148f554b8699492926254d35a98259de43a2d993993a8fa89e3ef92b48ac"
Nov 02 21:45:04.384345 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.384296    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" event={"ID":"987cde20-ab8f-4bce-a527-d57626938569","Type":"ContainerStarted","Data":"8e6284a10b8433074e370aa69646f3f2da49c9b98e766d27e28ce5a69dffc814"}
Nov 02 21:45:04.399194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.399021    4528 scope.go:122] "RemoveContainer" containerID="57c1b83c14e4bc079c231ff507ae640f88ac93d9fe6100c99b707f9d089ea589"
Nov 02 21:45:04.422165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.422140    4528 scope.go:122] "RemoveContainer" containerID="22cffabb583a32f7f828309b0146111efd3313439585d546ffdfbced5d3196a4"
Nov 02 21:45:04.435180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.435161    4528 scope.go:122] "RemoveContainer" containerID="444a3e295ba428050ef7f28446331526dfbdc8509bac24f6ec4e52fdc02078d5"
Nov 02 21:45:04.447257 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.447237    4528 scope.go:122] "RemoveContainer" containerID="19dffbb95d8a9ba3caa207fc5dd1cdcedc5f1f90d26d39b630366e6cc26f65f1"
Nov 02 21:45:04.459358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:04.459339    4528 scope.go:122] "RemoveContainer" containerID="6261682d8592861fe150960e5ac0ecf7aff9cb128a5e796e7c8d9681a46a9108"
Nov 02 21:45:05.392120 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.392089    4528 scope.go:122] "RemoveContainer" containerID="05d0b04e2128e432f64f89e2ade53027e53bbab708c0eb276d71c84e1470db6f"
Nov 02 21:45:05.392578 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:05.392476    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md_odh-ai-gateway-infra(a2763b03-c0d8-4c1d-8b46-8c6615749031)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031"
Nov 02 21:45:05.393926 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.393901    4528 scope.go:122] "RemoveContainer" containerID="626f79c1f95e184769beca6b9b4a20e4ddf59c823509c14c08058486048455d8"
Nov 02 21:45:05.394231 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:05.394205    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-5scx6_odh-ai-gateway-infra(453701cb-6a3f-48ae-8cb0-6831a58e06ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef"
Nov 02 21:45:05.395543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.395522    4528 scope.go:122] "RemoveContainer" containerID="4eae84897c6d37d26b956919e60a3d7660a3b79c6cfd0380662bc62ac2aac158"
Nov 02 21:45:05.395991 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:05.395963    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v_odh-ai-gateway-infra(938d59c5-d567-4af2-8672-e29bf28a458e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v" podUID="938d59c5-d567-4af2-8672-e29bf28a458e"
Nov 02 21:45:05.397803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.397781    4528 scope.go:122] "RemoveContainer" containerID="7777f07f3b1e9aad0c330b6887bb663d6d0a655544af66967056eb8a214ae1ee"
Nov 02 21:45:05.398053 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:05.398030    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk_odh-ai-gateway-infra(f3b328bc-d9e2-4821-8e0e-3366a1d84799)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799"
Nov 02 21:45:05.399375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.399359    4528 scope.go:122] "RemoveContainer" containerID="8a24775af519ede616c2aac5ea73405cba0d71a50f922df5d0d1e503d693e234"
Nov 02 21:45:05.399563 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:05.399544    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4_odh-ai-gateway-infra(99712198-d097-4fed-ba4a-76bfa3888fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" podUID="99712198-d097-4fed-ba4a-76bfa3888fee"
Nov 02 21:45:05.401028 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.401010    4528 scope.go:122] "RemoveContainer" containerID="26de1ef5926c80a65727e8ff00e305bc58938885a11d5ed4cf1cfd3011d289f5"
Nov 02 21:45:05.401273 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:05.401251    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4_odh-ai-gateway-infra(bec2d0e4-4b05-4e38-8865-d30dd5085528)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528"
Nov 02 21:45:05.402475 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.402459    4528 scope.go:122] "RemoveContainer" containerID="dd47cdba4f5c168c704e0f758eeccae87a80a9dc8f5a894b990b42f79796d720"
Nov 02 21:45:05.402647 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:05.402625    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf_odh-ai-gateway-infra(ad43ab3f-245e-4af2-9b9e-0de944ccc0ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf" podUID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce"
Nov 02 21:45:05.404034 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.404010    4528 generic.go:356] "Generic (PLEG): container finished" podID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerID="143cdcb3e0bd6e80d9cb0ee59ef25724a6fa612ac09c6fccb95e41768cff3ea5" exitCode=60
Nov 02 21:45:05.404149 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.404090    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" event={"ID":"2f8e4c51-545b-4043-8bbc-6314c1e91839","Type":"ContainerDied","Data":"143cdcb3e0bd6e80d9cb0ee59ef25724a6fa612ac09c6fccb95e41768cff3ea5"}
Nov 02 21:45:05.404227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.404193    4528 scope.go:122] "RemoveContainer" containerID="0524738e8295deaa70efd13905fd70e806a8df00d82e7a41cb3bbd438a0eb877"
Nov 02 21:45:05.404299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.404281    4528 scope.go:122] "RemoveContainer" containerID="143cdcb3e0bd6e80d9cb0ee59ef25724a6fa612ac09c6fccb95e41768cff3ea5"
Nov 02 21:45:05.404482 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:05.404460    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255_odh-ai-gateway-infra(2f8e4c51-545b-4043-8bbc-6314c1e91839)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839"
Nov 02 21:45:05.405919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.405897    4528 generic.go:356] "Generic (PLEG): container finished" podID="987cde20-ab8f-4bce-a527-d57626938569" containerID="8e6284a10b8433074e370aa69646f3f2da49c9b98e766d27e28ce5a69dffc814" exitCode=60
Nov 02 21:45:05.405995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.405942    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" event={"ID":"987cde20-ab8f-4bce-a527-d57626938569","Type":"ContainerDied","Data":"8e6284a10b8433074e370aa69646f3f2da49c9b98e766d27e28ce5a69dffc814"}
Nov 02 21:45:05.406152 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.406140    4528 scope.go:122] "RemoveContainer" containerID="8e6284a10b8433074e370aa69646f3f2da49c9b98e766d27e28ce5a69dffc814"
Nov 02 21:45:05.406318 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:05.406303    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4_odh-ai-gateway-infra(987cde20-ab8f-4bce-a527-d57626938569)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" podUID="987cde20-ab8f-4bce-a527-d57626938569"
Nov 02 21:45:05.416777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:05.416761    4528 scope.go:122] "RemoveContainer" containerID="e62c57e7bc92e23244d027ab043a326264d58e7b9733315eca4f3ed65dd2259c"
Nov 02 21:45:06.415575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:06.415528    4528 scope.go:122] "RemoveContainer" containerID="143cdcb3e0bd6e80d9cb0ee59ef25724a6fa612ac09c6fccb95e41768cff3ea5"
Nov 02 21:45:06.416006 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:06.415851    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255_odh-ai-gateway-infra(2f8e4c51-545b-4043-8bbc-6314c1e91839)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839"
Nov 02 21:45:06.417063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:06.417043    4528 scope.go:122] "RemoveContainer" containerID="8e6284a10b8433074e370aa69646f3f2da49c9b98e766d27e28ce5a69dffc814"
Nov 02 21:45:06.417307 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:06.417289    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4_odh-ai-gateway-infra(987cde20-ab8f-4bce-a527-d57626938569)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" podUID="987cde20-ab8f-4bce-a527-d57626938569"
Nov 02 21:45:08.033913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.033866    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"]
Nov 02 21:45:08.219843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.219822    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:08.323848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.323766    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"]
Nov 02 21:45:08.324021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.323987    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x" podUID="6052e3ec-2963-4b40-a013-d2fe85f7f989" containerName="payload-processing" containerID="cri-o://160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2" gracePeriod=30
Nov 02 21:45:08.391971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.391939    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"]
Nov 02 21:45:08.392139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.392043    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938d59c5-d567-4af2-8672-e29bf28a458e-kube-api-access-rkvp4\" (UniqueName: \"kubernetes.io/projected/938d59c5-d567-4af2-8672-e29bf28a458e-kube-api-access-rkvp4\") pod \"938d59c5-d567-4af2-8672-e29bf28a458e\" (UID: \"938d59c5-d567-4af2-8672-e29bf28a458e\") "
Nov 02 21:45:08.392198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.392128    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc" podUID="ab3df98c-f72b-4e36-b436-18bbcba2c20a" containerName="payload-pre-processing" containerID="cri-o://918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7" gracePeriod=30
Nov 02 21:45:08.392265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.392217    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938d59c5-d567-4af2-8672-e29bf28a458e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938d59c5-d567-4af2-8672-e29bf28a458e-maas-api-service-ca\") pod \"938d59c5-d567-4af2-8672-e29bf28a458e\" (UID: \"938d59c5-d567-4af2-8672-e29bf28a458e\") "
Nov 02 21:45:08.392641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.392614    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938d59c5-d567-4af2-8672-e29bf28a458e-maas-api-service-ca" pod "938d59c5-d567-4af2-8672-e29bf28a458e" (UID: "938d59c5-d567-4af2-8672-e29bf28a458e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:08.394242 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.394219    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938d59c5-d567-4af2-8672-e29bf28a458e-kube-api-access-rkvp4" pod "938d59c5-d567-4af2-8672-e29bf28a458e" (UID: "938d59c5-d567-4af2-8672-e29bf28a458e"). InnerVolumeSpecName "kube-api-access-rkvp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:08.437146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.435062    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"
Nov 02 21:45:08.437146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.435090    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v" event={"ID":"938d59c5-d567-4af2-8672-e29bf28a458e","Type":"ContainerDied","Data":"fb340b4edae81916252e1f147b70a214f6dc33aa7201c7736c49019b7482ca62"}
Nov 02 21:45:08.437146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.435174    4528 scope.go:122] "RemoveContainer" containerID="4eae84897c6d37d26b956919e60a3d7660a3b79c6cfd0380662bc62ac2aac158"
Nov 02 21:45:08.493624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.493592    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkvp4\" (UniqueName: \"kubernetes.io/projected/938d59c5-d567-4af2-8672-e29bf28a458e-kube-api-access-rkvp4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:08.493624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.493620    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938d59c5-d567-4af2-8672-e29bf28a458e-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:08.672141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.669345    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:45:08.689079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.689051    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:45:08.743235 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.742410    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"]
Nov 02 21:45:08.745442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.745418    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a9abc-29894265-mfm6v"]
Nov 02 21:45:08.797386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.797355    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6052e3ec-2963-4b40-a013-d2fe85f7f989-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052e3ec-2963-4b40-a013-d2fe85f7f989-plugins-config-volume\") pod \"6052e3ec-2963-4b40-a013-d2fe85f7f989\" (UID: \"6052e3ec-2963-4b40-a013-d2fe85f7f989\") "
Nov 02 21:45:08.797386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.797389    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3df98c-f72b-4e36-b436-18bbcba2c20a-kube-api-access-qffn7\" (UniqueName: \"kubernetes.io/projected/ab3df98c-f72b-4e36-b436-18bbcba2c20a-kube-api-access-qffn7\") pod \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\" (UID: \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\") "
Nov 02 21:45:08.797601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.797454    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6052e3ec-2963-4b40-a013-d2fe85f7f989-kube-api-access-9dwn8\" (UniqueName: \"kubernetes.io/projected/6052e3ec-2963-4b40-a013-d2fe85f7f989-kube-api-access-9dwn8\") pod \"6052e3ec-2963-4b40-a013-d2fe85f7f989\" (UID: \"6052e3ec-2963-4b40-a013-d2fe85f7f989\") "
Nov 02 21:45:08.797661 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.797616    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3df98c-f72b-4e36-b436-18bbcba2c20a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3df98c-f72b-4e36-b436-18bbcba2c20a-plugins-config-volume\") pod \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\" (UID: \"ab3df98c-f72b-4e36-b436-18bbcba2c20a\") "
Nov 02 21:45:08.797754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.797721    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6052e3ec-2963-4b40-a013-d2fe85f7f989-plugins-config-volume" pod "6052e3ec-2963-4b40-a013-d2fe85f7f989" (UID: "6052e3ec-2963-4b40-a013-d2fe85f7f989"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:08.798028 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.798003    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3df98c-f72b-4e36-b436-18bbcba2c20a-plugins-config-volume" pod "ab3df98c-f72b-4e36-b436-18bbcba2c20a" (UID: "ab3df98c-f72b-4e36-b436-18bbcba2c20a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:08.798191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.798175    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3df98c-f72b-4e36-b436-18bbcba2c20a-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:08.798263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.798197    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052e3ec-2963-4b40-a013-d2fe85f7f989-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:08.799651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.799628    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6052e3ec-2963-4b40-a013-d2fe85f7f989-kube-api-access-9dwn8" pod "6052e3ec-2963-4b40-a013-d2fe85f7f989" (UID: "6052e3ec-2963-4b40-a013-d2fe85f7f989"). InnerVolumeSpecName "kube-api-access-9dwn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:08.799651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.799644    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3df98c-f72b-4e36-b436-18bbcba2c20a-kube-api-access-qffn7" pod "ab3df98c-f72b-4e36-b436-18bbcba2c20a" (UID: "ab3df98c-f72b-4e36-b436-18bbcba2c20a"). InnerVolumeSpecName "kube-api-access-qffn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:08.907181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.906263    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dwn8\" (UniqueName: \"kubernetes.io/projected/6052e3ec-2963-4b40-a013-d2fe85f7f989-kube-api-access-9dwn8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:08.907181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.906313    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qffn7\" (UniqueName: \"kubernetes.io/projected/ab3df98c-f72b-4e36-b436-18bbcba2c20a-kube-api-access-qffn7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:08.951067 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:08.950965    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938d59c5-d567-4af2-8672-e29bf28a458e" path="/var/lib/kubelet/pods/938d59c5-d567-4af2-8672-e29bf28a458e/volumes"
Nov 02 21:45:09.366572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.366533    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"]
Nov 02 21:45:09.453537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.451952    4528 generic.go:356] "Generic (PLEG): container finished" podID="ab3df98c-f72b-4e36-b436-18bbcba2c20a" containerID="918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7" exitCode=0
Nov 02 21:45:09.453537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.452036    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc" event={"ID":"ab3df98c-f72b-4e36-b436-18bbcba2c20a","Type":"ContainerDied","Data":"918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7"}
Nov 02 21:45:09.453537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.452083    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc" event={"ID":"ab3df98c-f72b-4e36-b436-18bbcba2c20a","Type":"ContainerDied","Data":"3e10f4064e5f1b5c6f966da45f6915fcbda94f721ba816951a192bee60932e4b"}
Nov 02 21:45:09.453537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.452124    4528 scope.go:122] "RemoveContainer" containerID="918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7"
Nov 02 21:45:09.453537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.452226    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"
Nov 02 21:45:09.456431 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.456402    4528 generic.go:356] "Generic (PLEG): container finished" podID="6052e3ec-2963-4b40-a013-d2fe85f7f989" containerID="160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2" exitCode=0
Nov 02 21:45:09.456531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.456461    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"
Nov 02 21:45:09.456531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.456470    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x" event={"ID":"6052e3ec-2963-4b40-a013-d2fe85f7f989","Type":"ContainerDied","Data":"160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2"}
Nov 02 21:45:09.456531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.456509    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x" event={"ID":"6052e3ec-2963-4b40-a013-d2fe85f7f989","Type":"ContainerDied","Data":"aa3f56bf9dbcbbc5c1d587681785c4ea8b4498166ad93a7fb52f198d369c9da1"}
Nov 02 21:45:09.471160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.470917    4528 scope.go:122] "RemoveContainer" containerID="918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7"
Nov 02 21:45:09.471403 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:09.471372    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7\": container with ID starting with 918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7 not found: ID does not exist" containerID="918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7"
Nov 02 21:45:09.471498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.471410    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7"} err="failed to get container status \"918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7\": rpc error: code = NotFound desc = could not find container \"918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7\": container with ID starting with 918866039bfb28b5ac7d272a7648707ab8a3080749be98fe221181f4b35728d7 not found: ID does not exist"
Nov 02 21:45:09.471498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.471433    4528 scope.go:122] "RemoveContainer" containerID="160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2"
Nov 02 21:45:09.493556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.493525    4528 scope.go:122] "RemoveContainer" containerID="160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2"
Nov 02 21:45:09.493901 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:09.493882    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2\": container with ID starting with 160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2 not found: ID does not exist" containerID="160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2"
Nov 02 21:45:09.493986 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.493912    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2"} err="failed to get container status \"160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2\": rpc error: code = NotFound desc = could not find container \"160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2\": container with ID starting with 160e28022d563e5d2e79718d274ffe37f2867ffdf3b660d5d3c0a3f9c8c6fcb2 not found: ID does not exist"
Nov 02 21:45:09.580782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.580760    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:09.604387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.604355    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"]
Nov 02 21:45:09.607239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.607216    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a9abc-7878f4c54c-p77sc"]
Nov 02 21:45:09.721316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.717672    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-maas-api-service-ca\") pod \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\" (UID: \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\") "
Nov 02 21:45:09.721316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.717769    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-kube-api-access-xd88k\" (UniqueName: \"kubernetes.io/projected/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-kube-api-access-xd88k\") pod \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\" (UID: \"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce\") "
Nov 02 21:45:09.721316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.719204    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-maas-api-service-ca" pod "ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" (UID: "ad43ab3f-245e-4af2-9b9e-0de944ccc0ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:09.728758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.728722    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-kube-api-access-xd88k" pod "ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" (UID: "ad43ab3f-245e-4af2-9b9e-0de944ccc0ce"). InnerVolumeSpecName "kube-api-access-xd88k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:09.731500 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.731470    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"]
Nov 02 21:45:09.736994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.736966    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a9abc-97cfc45b-dzh5x"]
Nov 02 21:45:09.797288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.797258    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"]
Nov 02 21:45:09.797496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.797466    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt" podUID="c6d5b1cb-44dc-4659-8ac4-38e5d115cad0" containerName="payload-processing" containerID="cri-o://8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1" gracePeriod=30
Nov 02 21:45:09.819709 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.819688    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd88k\" (UniqueName: \"kubernetes.io/projected/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-kube-api-access-xd88k\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:09.819789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.819712    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:09.884856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.884822    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"]
Nov 02 21:45:09.885087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:09.885063    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq" podUID="3aac0858-7c8f-4144-b5b5-ed9389162e81" containerName="payload-pre-processing" containerID="cri-o://2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb" gracePeriod=30
Nov 02 21:45:10.153414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.153384    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:45:10.180880 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.180853    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:45:10.331161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.331135    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aac0858-7c8f-4144-b5b5-ed9389162e81-kube-api-access-75xkh\" (UniqueName: \"kubernetes.io/projected/3aac0858-7c8f-4144-b5b5-ed9389162e81-kube-api-access-75xkh\") pod \"3aac0858-7c8f-4144-b5b5-ed9389162e81\" (UID: \"3aac0858-7c8f-4144-b5b5-ed9389162e81\") "
Nov 02 21:45:10.331330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.331181    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-kube-api-access-vf2ks\" (UniqueName: \"kubernetes.io/projected/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-kube-api-access-vf2ks\") pod \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\" (UID: \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\") "
Nov 02 21:45:10.331330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.331216    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aac0858-7c8f-4144-b5b5-ed9389162e81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aac0858-7c8f-4144-b5b5-ed9389162e81-plugins-config-volume\") pod \"3aac0858-7c8f-4144-b5b5-ed9389162e81\" (UID: \"3aac0858-7c8f-4144-b5b5-ed9389162e81\") "
Nov 02 21:45:10.331466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.331348    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-plugins-config-volume\") pod \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\" (UID: \"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0\") "
Nov 02 21:45:10.331727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.331676    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aac0858-7c8f-4144-b5b5-ed9389162e81-plugins-config-volume" pod "3aac0858-7c8f-4144-b5b5-ed9389162e81" (UID: "3aac0858-7c8f-4144-b5b5-ed9389162e81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:10.331727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.331690    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-plugins-config-volume" pod "c6d5b1cb-44dc-4659-8ac4-38e5d115cad0" (UID: "c6d5b1cb-44dc-4659-8ac4-38e5d115cad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:10.333283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.333261    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aac0858-7c8f-4144-b5b5-ed9389162e81-kube-api-access-75xkh" pod "3aac0858-7c8f-4144-b5b5-ed9389162e81" (UID: "3aac0858-7c8f-4144-b5b5-ed9389162e81"). InnerVolumeSpecName "kube-api-access-75xkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:10.333397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.333368    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-kube-api-access-vf2ks" pod "c6d5b1cb-44dc-4659-8ac4-38e5d115cad0" (UID: "c6d5b1cb-44dc-4659-8ac4-38e5d115cad0"). InnerVolumeSpecName "kube-api-access-vf2ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:10.464354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.446154    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75xkh\" (UniqueName: \"kubernetes.io/projected/3aac0858-7c8f-4144-b5b5-ed9389162e81-kube-api-access-75xkh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:10.464919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.464875    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf2ks\" (UniqueName: \"kubernetes.io/projected/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-kube-api-access-vf2ks\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:10.464919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.464913    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aac0858-7c8f-4144-b5b5-ed9389162e81-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:10.465132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.464928    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:10.472692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.472663    4528 generic.go:356] "Generic (PLEG): container finished" podID="c6d5b1cb-44dc-4659-8ac4-38e5d115cad0" containerID="8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1" exitCode=0
Nov 02 21:45:10.472795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.472766    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt" event={"ID":"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0","Type":"ContainerDied","Data":"8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1"}
Nov 02 21:45:10.472842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.472830    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt" event={"ID":"c6d5b1cb-44dc-4659-8ac4-38e5d115cad0","Type":"ContainerDied","Data":"5664aa07106b7ba16103aca48858f98f5cfa7760f22fb4a90ee6f1ffcfba0601"}
Nov 02 21:45:10.472892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.472855    4528 scope.go:122] "RemoveContainer" containerID="8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1"
Nov 02 21:45:10.472971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.472961    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"
Nov 02 21:45:10.477676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.477602    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf" event={"ID":"ad43ab3f-245e-4af2-9b9e-0de944ccc0ce","Type":"ContainerDied","Data":"67872d5d7bca527b56855d6d9b0b5752a99591a3640b33a062d7544a307642d9"}
Nov 02 21:45:10.477875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.477855    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"
Nov 02 21:45:10.482652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.482631    4528 generic.go:356] "Generic (PLEG): container finished" podID="3aac0858-7c8f-4144-b5b5-ed9389162e81" containerID="2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb" exitCode=0
Nov 02 21:45:10.482790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.482775    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq" event={"ID":"3aac0858-7c8f-4144-b5b5-ed9389162e81","Type":"ContainerDied","Data":"2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb"}
Nov 02 21:45:10.482903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.482879    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq" event={"ID":"3aac0858-7c8f-4144-b5b5-ed9389162e81","Type":"ContainerDied","Data":"1a7ca55402160bbff9a812846cbcf105ec8dafaa7b8d801f34ae89bacd5b7e67"}
Nov 02 21:45:10.483082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.483066    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"
Nov 02 21:45:10.486313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.486271    4528 scope.go:122] "RemoveContainer" containerID="8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1"
Nov 02 21:45:10.486565 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:10.486542    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1\": container with ID starting with 8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1 not found: ID does not exist" containerID="8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1"
Nov 02 21:45:10.486643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.486576    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1"} err="failed to get container status \"8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1\": rpc error: code = NotFound desc = could not find container \"8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1\": container with ID starting with 8e854c2006210a77b1f75fe2a962c6dd6f767351fbea6cb0baf6b12e6045d0e1 not found: ID does not exist"
Nov 02 21:45:10.486643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.486601    4528 scope.go:122] "RemoveContainer" containerID="dd47cdba4f5c168c704e0f758eeccae87a80a9dc8f5a894b990b42f79796d720"
Nov 02 21:45:10.496247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.496233    4528 scope.go:122] "RemoveContainer" containerID="2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb"
Nov 02 21:45:10.504812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.504797    4528 scope.go:122] "RemoveContainer" containerID="2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb"
Nov 02 21:45:10.505142 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:10.505094    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb\": container with ID starting with 2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb not found: ID does not exist" containerID="2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb"
Nov 02 21:45:10.505211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.505152    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb"} err="failed to get container status \"2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb\": rpc error: code = NotFound desc = could not find container \"2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb\": container with ID starting with 2dcbbcf235182307b0c053fd54614f1d4cf2284f11ae2489c20a534cc347ecfb not found: ID does not exist"
Nov 02 21:45:10.638044 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.637966    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"]
Nov 02 21:45:10.641250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.641228    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d3617-5cff4ffb4c-gw4sq"]
Nov 02 21:45:10.750471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.750445    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"]
Nov 02 21:45:10.753759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.753731    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d3617-6859b46f66-ct5nt"]
Nov 02 21:45:10.862445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.862416    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"]
Nov 02 21:45:10.865027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.865006    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7d3617-29894265-dn5mf"]
Nov 02 21:45:10.914687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.914616    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aac0858-7c8f-4144-b5b5-ed9389162e81" path="/var/lib/kubelet/pods/3aac0858-7c8f-4144-b5b5-ed9389162e81/volumes"
Nov 02 21:45:10.915158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.915136    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6052e3ec-2963-4b40-a013-d2fe85f7f989" path="/var/lib/kubelet/pods/6052e3ec-2963-4b40-a013-d2fe85f7f989/volumes"
Nov 02 21:45:10.915545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.915528    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3df98c-f72b-4e36-b436-18bbcba2c20a" path="/var/lib/kubelet/pods/ab3df98c-f72b-4e36-b436-18bbcba2c20a/volumes"
Nov 02 21:45:10.915910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.915893    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" path="/var/lib/kubelet/pods/ad43ab3f-245e-4af2-9b9e-0de944ccc0ce/volumes"
Nov 02 21:45:10.916601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:10.916581    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d5b1cb-44dc-4659-8ac4-38e5d115cad0" path="/var/lib/kubelet/pods/c6d5b1cb-44dc-4659-8ac4-38e5d115cad0/volumes"
Nov 02 21:45:12.953613 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:12.953584    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"]
Nov 02 21:45:12.954003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:12.953764    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy" containerID="cri-o://8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d" gracePeriod=30
Nov 02 21:45:13.906886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:13.906854    4528 scope.go:122] "RemoveContainer" containerID="8a24775af519ede616c2aac5ea73405cba0d71a50f922df5d0d1e503d693e234"
Nov 02 21:45:13.907069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:13.906997    4528 scope.go:122] "RemoveContainer" containerID="7777f07f3b1e9aad0c330b6887bb663d6d0a655544af66967056eb8a214ae1ee"
Nov 02 21:45:13.907209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:13.907192    4528 scope.go:122] "RemoveContainer" containerID="626f79c1f95e184769beca6b9b4a20e4ddf59c823509c14c08058486048455d8"
Nov 02 21:45:14.113294 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:14.113262    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:45:14.516026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.515998    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" event={"ID":"453701cb-6a3f-48ae-8cb0-6831a58e06ef","Type":"ContainerStarted","Data":"7849a10a43b9983f4a55b7a0ed683e758c5126a7ec5e57e404f9f989e4da15f8"}
Nov 02 21:45:14.517954 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.517931    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" event={"ID":"f3b328bc-d9e2-4821-8e0e-3366a1d84799","Type":"ContainerStarted","Data":"69dbf3163e79575bb0aba476ae2dfc0181644d1afccea927d9568ed7ea9d4a13"}
Nov 02 21:45:14.519667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.519650    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" event={"ID":"99712198-d097-4fed-ba4a-76bfa3888fee","Type":"ContainerStarted","Data":"2e9ce0fc7945270244ed88f94ecadbefa351d5487de79f523e66cbfa8083712e"}
Nov 02 21:45:14.906214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.906178    4528 scope.go:122] "RemoveContainer" containerID="8e6284a10b8433074e370aa69646f3f2da49c9b98e766d27e28ce5a69dffc814"
Nov 02 21:45:14.952675 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.952647    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"]
Nov 02 21:45:14.953288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953270    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" containerName="cleanup"
Nov 02 21:45:14.953395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953289    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" containerName="cleanup"
Nov 02 21:45:14.953395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953327    4528 cpu_manager.go:414] "containerMap: removing container" podUID="938d59c5-d567-4af2-8672-e29bf28a458e" containerName="cleanup"
Nov 02 21:45:14.953395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953335    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938d59c5-d567-4af2-8672-e29bf28a458e" containerName="cleanup"
Nov 02 21:45:14.953395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953350    4528 cpu_manager.go:414] "containerMap: removing container" podUID="938d59c5-d567-4af2-8672-e29bf28a458e" containerName="cleanup"
Nov 02 21:45:14.953395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953358    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938d59c5-d567-4af2-8672-e29bf28a458e" containerName="cleanup"
Nov 02 21:45:14.953395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953390    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c6d5b1cb-44dc-4659-8ac4-38e5d115cad0" containerName="payload-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953398    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d5b1cb-44dc-4659-8ac4-38e5d115cad0" containerName="payload-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953416    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3aac0858-7c8f-4144-b5b5-ed9389162e81" containerName="payload-pre-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953424    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aac0858-7c8f-4144-b5b5-ed9389162e81" containerName="payload-pre-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953439    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ab3df98c-f72b-4e36-b436-18bbcba2c20a" containerName="payload-pre-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953447    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3df98c-f72b-4e36-b436-18bbcba2c20a" containerName="payload-pre-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953466    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6052e3ec-2963-4b40-a013-d2fe85f7f989" containerName="payload-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953473    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6052e3ec-2963-4b40-a013-d2fe85f7f989" containerName="payload-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953613    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d5b1cb-44dc-4659-8ac4-38e5d115cad0" containerName="payload-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953627    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aac0858-7c8f-4144-b5b5-ed9389162e81" containerName="payload-pre-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953640    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3df98c-f72b-4e36-b436-18bbcba2c20a" containerName="payload-pre-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953650    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6052e3ec-2963-4b40-a013-d2fe85f7f989" containerName="payload-processing"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953662    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" containerName="cleanup"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953675    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="938d59c5-d567-4af2-8672-e29bf28a458e" containerName="cleanup"
Nov 02 21:45:14.953749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.953683    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="938d59c5-d567-4af2-8672-e29bf28a458e" containerName="cleanup"
Nov 02 21:45:14.957432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:14.957414    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.032409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.026825    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-credential-socket\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.032409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.026888    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-certs\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.032409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.026964    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-istio-token\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.032409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.026993    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-envoy\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.032409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.027166    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-socket\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.032409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.027208    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpn8\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-kube-api-access-wvpn8\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.032409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.027240    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf178a9-eb59-4865-be00-c22072fe8439-istio-podinfo\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.032409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.027310    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-data\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.032409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.027364    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf178a9-eb59-4865-be00-c22072fe8439-istiod-ca-cert\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.095005 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.093593    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-8e90da-openshift-default-dockercfg-4jhcb\""
Nov 02 21:45:15.100281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.100173    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"]
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.138178    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-credential-socket\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.138281    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-certs\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.138369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-istio-token\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.138400    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-envoy\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.138499    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-socket\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.138548    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpn8\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-kube-api-access-wvpn8\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.138588    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf178a9-eb59-4865-be00-c22072fe8439-istio-podinfo\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.138657    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-data\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.138706    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf178a9-eb59-4865-be00-c22072fe8439-istiod-ca-cert\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.139562    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf178a9-eb59-4865-be00-c22072fe8439-istiod-ca-cert\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.139842    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-credential-socket\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.140081    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-certs\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.145195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.142693    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-socket\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.162813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.149038    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf178a9-eb59-4865-be00-c22072fe8439-istio-podinfo\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.162813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.150070    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-data\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.175748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.175716    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-envoy\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.199486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.199454    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-istio-token\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.199620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.199598    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpn8\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-kube-api-access-wvpn8\") pod \"e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") " pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.412415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.412391    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:15.528466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.528429    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" event={"ID":"987cde20-ab8f-4bce-a527-d57626938569","Type":"ContainerStarted","Data":"711d2cff467be22749e3fb8cc81bc7028ba83397e525c11307db7a0ebdd514b6"}
Nov 02 21:45:15.530379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.530353    4528 generic.go:356] "Generic (PLEG): container finished" podID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerID="7849a10a43b9983f4a55b7a0ed683e758c5126a7ec5e57e404f9f989e4da15f8" exitCode=60
Nov 02 21:45:15.530534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.530404    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" event={"ID":"453701cb-6a3f-48ae-8cb0-6831a58e06ef","Type":"ContainerDied","Data":"7849a10a43b9983f4a55b7a0ed683e758c5126a7ec5e57e404f9f989e4da15f8"}
Nov 02 21:45:15.530534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.530450    4528 scope.go:122] "RemoveContainer" containerID="626f79c1f95e184769beca6b9b4a20e4ddf59c823509c14c08058486048455d8"
Nov 02 21:45:15.530809 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.530791    4528 scope.go:122] "RemoveContainer" containerID="7849a10a43b9983f4a55b7a0ed683e758c5126a7ec5e57e404f9f989e4da15f8"
Nov 02 21:45:15.531173 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:15.531147    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-5scx6_odh-ai-gateway-infra(453701cb-6a3f-48ae-8cb0-6831a58e06ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef"
Nov 02 21:45:15.532992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.532969    4528 generic.go:356] "Generic (PLEG): container finished" podID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerID="69dbf3163e79575bb0aba476ae2dfc0181644d1afccea927d9568ed7ea9d4a13" exitCode=60
Nov 02 21:45:15.533647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.533163    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" event={"ID":"f3b328bc-d9e2-4821-8e0e-3366a1d84799","Type":"ContainerDied","Data":"69dbf3163e79575bb0aba476ae2dfc0181644d1afccea927d9568ed7ea9d4a13"}
Nov 02 21:45:15.533647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.533480    4528 scope.go:122] "RemoveContainer" containerID="69dbf3163e79575bb0aba476ae2dfc0181644d1afccea927d9568ed7ea9d4a13"
Nov 02 21:45:15.533813 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:15.533687    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk_odh-ai-gateway-infra(f3b328bc-d9e2-4821-8e0e-3366a1d84799)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799"
Nov 02 21:45:15.535614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.535593    4528 generic.go:356] "Generic (PLEG): container finished" podID="99712198-d097-4fed-ba4a-76bfa3888fee" containerID="2e9ce0fc7945270244ed88f94ecadbefa351d5487de79f523e66cbfa8083712e" exitCode=60
Nov 02 21:45:15.535714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.535650    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" event={"ID":"99712198-d097-4fed-ba4a-76bfa3888fee","Type":"ContainerDied","Data":"2e9ce0fc7945270244ed88f94ecadbefa351d5487de79f523e66cbfa8083712e"}
Nov 02 21:45:15.535961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.535947    4528 scope.go:122] "RemoveContainer" containerID="2e9ce0fc7945270244ed88f94ecadbefa351d5487de79f523e66cbfa8083712e"
Nov 02 21:45:15.536177 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:15.536159    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4_odh-ai-gateway-infra(99712198-d097-4fed-ba4a-76bfa3888fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" podUID="99712198-d097-4fed-ba4a-76bfa3888fee"
Nov 02 21:45:15.545564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.545524    4528 scope.go:122] "RemoveContainer" containerID="7777f07f3b1e9aad0c330b6887bb663d6d0a655544af66967056eb8a214ae1ee"
Nov 02 21:45:15.557918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.557896    4528 scope.go:122] "RemoveContainer" containerID="8a24775af519ede616c2aac5ea73405cba0d71a50f922df5d0d1e503d693e234"
Nov 02 21:45:15.646811 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.646784    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"]
Nov 02 21:45:15.754218 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.754143    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"]
Nov 02 21:45:15.828857 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:15.828828    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf178a9_eb59_4865_be00_c22072fe8439.slice/crio-7e8a78884a79e8385330f77665d6cf2b26daf4127896afe37597e6ec4937b84d WatchSource:0}: Error finding container 7e8a78884a79e8385330f77665d6cf2b26daf4127896afe37597e6ec4937b84d: Status 404 returned error can't find the container with id 7e8a78884a79e8385330f77665d6cf2b26daf4127896afe37597e6ec4937b84d
Nov 02 21:45:15.831686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.831655    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:45:15.831797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.831718    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:45:15.831797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.831747    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:45:15.832558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.832535    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"]
Nov 02 21:45:15.867321 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.867296    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"]
Nov 02 21:45:15.906730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.906711    4528 scope.go:122] "RemoveContainer" containerID="143cdcb3e0bd6e80d9cb0ee59ef25724a6fa612ac09c6fccb95e41768cff3ea5"
Nov 02 21:45:15.906930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:15.906914    4528 scope.go:122] "RemoveContainer" containerID="05d0b04e2128e432f64f89e2ade53027e53bbab708c0eb276d71c84e1470db6f"
Nov 02 21:45:16.546290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.546261    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" event={"ID":"2f8e4c51-545b-4043-8bbc-6314c1e91839","Type":"ContainerStarted","Data":"2cc569d8aa33cbb2624dff5a85a9f25b349aa6bb510024a80b2d570fe82ee7c7"}
Nov 02 21:45:16.548086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.548064    4528 generic.go:356] "Generic (PLEG): container finished" podID="987cde20-ab8f-4bce-a527-d57626938569" containerID="711d2cff467be22749e3fb8cc81bc7028ba83397e525c11307db7a0ebdd514b6" exitCode=60
Nov 02 21:45:16.548217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.548136    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" event={"ID":"987cde20-ab8f-4bce-a527-d57626938569","Type":"ContainerDied","Data":"711d2cff467be22749e3fb8cc81bc7028ba83397e525c11307db7a0ebdd514b6"}
Nov 02 21:45:16.548217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.548193    4528 scope.go:122] "RemoveContainer" containerID="8e6284a10b8433074e370aa69646f3f2da49c9b98e766d27e28ce5a69dffc814"
Nov 02 21:45:16.548545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.548530    4528 scope.go:122] "RemoveContainer" containerID="711d2cff467be22749e3fb8cc81bc7028ba83397e525c11307db7a0ebdd514b6"
Nov 02 21:45:16.548855 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:16.548833    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4_odh-ai-gateway-infra(987cde20-ab8f-4bce-a527-d57626938569)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" podUID="987cde20-ab8f-4bce-a527-d57626938569"
Nov 02 21:45:16.549734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.549706    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" event={"ID":"6bf178a9-eb59-4865-be00-c22072fe8439","Type":"ContainerStarted","Data":"ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf"}
Nov 02 21:45:16.549838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.549753    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" event={"ID":"6bf178a9-eb59-4865-be00-c22072fe8439","Type":"ContainerStarted","Data":"7e8a78884a79e8385330f77665d6cf2b26daf4127896afe37597e6ec4937b84d"}
Nov 02 21:45:16.551549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.551531    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" event={"ID":"a2763b03-c0d8-4c1d-8b46-8c6615749031","Type":"ContainerStarted","Data":"3f1b1104f67599d2a489389ad8128047c64afc1ee255719b6ab8a8b7ac02ce36"}
Nov 02 21:45:16.658252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.658220    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"]
Nov 02 21:45:16.857370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.857300    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:16.860273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.860253    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:16.863645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.863629    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:16.906297 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.906274    4528 scope.go:122] "RemoveContainer" containerID="26de1ef5926c80a65727e8ff00e305bc58938885a11d5ed4cf1cfd3011d289f5"
Nov 02 21:45:16.958410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.958387    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99712198-d097-4fed-ba4a-76bfa3888fee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99712198-d097-4fed-ba4a-76bfa3888fee-maas-api-service-ca\") pod \"99712198-d097-4fed-ba4a-76bfa3888fee\" (UID: \"99712198-d097-4fed-ba4a-76bfa3888fee\") "
Nov 02 21:45:16.958520 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.958441    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b328bc-d9e2-4821-8e0e-3366a1d84799-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b328bc-d9e2-4821-8e0e-3366a1d84799-maas-api-service-ca\") pod \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\" (UID: \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\") "
Nov 02 21:45:16.958520 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.958476    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453701cb-6a3f-48ae-8cb0-6831a58e06ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453701cb-6a3f-48ae-8cb0-6831a58e06ef-maas-api-service-ca\") pod \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\" (UID: \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\") "
Nov 02 21:45:16.958650 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.958563    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99712198-d097-4fed-ba4a-76bfa3888fee-kube-api-access-c9jtx\" (UniqueName: \"kubernetes.io/projected/99712198-d097-4fed-ba4a-76bfa3888fee-kube-api-access-c9jtx\") pod \"99712198-d097-4fed-ba4a-76bfa3888fee\" (UID: \"99712198-d097-4fed-ba4a-76bfa3888fee\") "
Nov 02 21:45:16.958711 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.958648    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453701cb-6a3f-48ae-8cb0-6831a58e06ef-kube-api-access-wxpwz\" (UniqueName: \"kubernetes.io/projected/453701cb-6a3f-48ae-8cb0-6831a58e06ef-kube-api-access-wxpwz\") pod \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\" (UID: \"453701cb-6a3f-48ae-8cb0-6831a58e06ef\") "
Nov 02 21:45:16.958711 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.958679    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b328bc-d9e2-4821-8e0e-3366a1d84799-kube-api-access-khlhh\" (UniqueName: \"kubernetes.io/projected/f3b328bc-d9e2-4821-8e0e-3366a1d84799-kube-api-access-khlhh\") pod \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\" (UID: \"f3b328bc-d9e2-4821-8e0e-3366a1d84799\") "
Nov 02 21:45:16.958839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.958745    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99712198-d097-4fed-ba4a-76bfa3888fee-maas-api-service-ca" pod "99712198-d097-4fed-ba4a-76bfa3888fee" (UID: "99712198-d097-4fed-ba4a-76bfa3888fee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:16.958906 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.958830    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b328bc-d9e2-4821-8e0e-3366a1d84799-maas-api-service-ca" pod "f3b328bc-d9e2-4821-8e0e-3366a1d84799" (UID: "f3b328bc-d9e2-4821-8e0e-3366a1d84799"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:16.958960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.958918    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453701cb-6a3f-48ae-8cb0-6831a58e06ef-maas-api-service-ca" pod "453701cb-6a3f-48ae-8cb0-6831a58e06ef" (UID: "453701cb-6a3f-48ae-8cb0-6831a58e06ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:16.959208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.959182    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b328bc-d9e2-4821-8e0e-3366a1d84799-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:16.959313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.959211    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453701cb-6a3f-48ae-8cb0-6831a58e06ef-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:16.959313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.959226    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99712198-d097-4fed-ba4a-76bfa3888fee-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:16.960673 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.960655    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99712198-d097-4fed-ba4a-76bfa3888fee-kube-api-access-c9jtx" pod "99712198-d097-4fed-ba4a-76bfa3888fee" (UID: "99712198-d097-4fed-ba4a-76bfa3888fee"). InnerVolumeSpecName "kube-api-access-c9jtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:16.960742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.960705    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b328bc-d9e2-4821-8e0e-3366a1d84799-kube-api-access-khlhh" pod "f3b328bc-d9e2-4821-8e0e-3366a1d84799" (UID: "f3b328bc-d9e2-4821-8e0e-3366a1d84799"). InnerVolumeSpecName "kube-api-access-khlhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:16.961059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:16.961034    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453701cb-6a3f-48ae-8cb0-6831a58e06ef-kube-api-access-wxpwz" pod "453701cb-6a3f-48ae-8cb0-6831a58e06ef" (UID: "453701cb-6a3f-48ae-8cb0-6831a58e06ef"). InnerVolumeSpecName "kube-api-access-wxpwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:17.060706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.060678    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jtx\" (UniqueName: \"kubernetes.io/projected/99712198-d097-4fed-ba4a-76bfa3888fee-kube-api-access-c9jtx\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:17.060706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.060704    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxpwz\" (UniqueName: \"kubernetes.io/projected/453701cb-6a3f-48ae-8cb0-6831a58e06ef-kube-api-access-wxpwz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:17.060859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.060714    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khlhh\" (UniqueName: \"kubernetes.io/projected/f3b328bc-d9e2-4821-8e0e-3366a1d84799-kube-api-access-khlhh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:17.240427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.240382    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" podStartSLOduration=3.240368127 podStartE2EDuration="3.240368127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:17.239393378 +0000 UTC m=+2204.502929694" watchObservedRunningTime="2026-11-02 21:45:17.240368127 +0000 UTC m=+2204.503904444"
Nov 02 21:45:17.412725 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.412648    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:17.418209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.418185    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:17.523898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.523866    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"]
Nov 02 21:45:17.524143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.524120    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy" containerID="cri-o://102c0939bd138096ad2b395c599010edb1c5aacab26a9d7f59a7ccef12e2d613" gracePeriod=30
Nov 02 21:45:17.562573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.562551    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"
Nov 02 21:45:17.562573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.562563    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6" event={"ID":"453701cb-6a3f-48ae-8cb0-6831a58e06ef","Type":"ContainerDied","Data":"a086cfc37e836cfc45cd43d841d2d3c852442180d1cd8c8ec8e437c752778947"}
Nov 02 21:45:17.563058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.562636    4528 scope.go:122] "RemoveContainer" containerID="7849a10a43b9983f4a55b7a0ed683e758c5126a7ec5e57e404f9f989e4da15f8"
Nov 02 21:45:17.564042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.564019    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"
Nov 02 21:45:17.564187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.564054    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk" event={"ID":"f3b328bc-d9e2-4821-8e0e-3366a1d84799","Type":"ContainerDied","Data":"33f77e58e8154aadcb2e232c80a8d15c956ed525aacf926d5f5b996ed52c586a"}
Nov 02 21:45:17.565773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.565754    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"
Nov 02 21:45:17.565980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.565790    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4" event={"ID":"99712198-d097-4fed-ba4a-76bfa3888fee","Type":"ContainerDied","Data":"87f1d4019e3ca2c2f4d66777607a4558d8a1dac36b9c3adddd6305c135b7dc8f"}
Nov 02 21:45:17.568099 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.568074    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" event={"ID":"bec2d0e4-4b05-4e38-8865-d30dd5085528","Type":"ContainerStarted","Data":"5464e91d6b9c81f8b6d6ecb68a49fe9ff1522ed486a50c0398d1581bb83ac1bc"}
Nov 02 21:45:17.570374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.570353    4528 generic.go:356] "Generic (PLEG): container finished" podID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerID="2cc569d8aa33cbb2624dff5a85a9f25b349aa6bb510024a80b2d570fe82ee7c7" exitCode=60
Nov 02 21:45:17.570472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.570415    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" event={"ID":"2f8e4c51-545b-4043-8bbc-6314c1e91839","Type":"ContainerDied","Data":"2cc569d8aa33cbb2624dff5a85a9f25b349aa6bb510024a80b2d570fe82ee7c7"}
Nov 02 21:45:17.570756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.570740    4528 scope.go:122] "RemoveContainer" containerID="2cc569d8aa33cbb2624dff5a85a9f25b349aa6bb510024a80b2d570fe82ee7c7"
Nov 02 21:45:17.570999 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:17.570976    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255_odh-ai-gateway-infra(2f8e4c51-545b-4043-8bbc-6314c1e91839)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839"
Nov 02 21:45:17.575898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.575882    4528 scope.go:122] "RemoveContainer" containerID="69dbf3163e79575bb0aba476ae2dfc0181644d1afccea927d9568ed7ea9d4a13"
Nov 02 21:45:17.577704 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.577684    4528 generic.go:356] "Generic (PLEG): container finished" podID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerID="3f1b1104f67599d2a489389ad8128047c64afc1ee255719b6ab8a8b7ac02ce36" exitCode=60
Nov 02 21:45:17.577802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.577747    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" event={"ID":"a2763b03-c0d8-4c1d-8b46-8c6615749031","Type":"ContainerDied","Data":"3f1b1104f67599d2a489389ad8128047c64afc1ee255719b6ab8a8b7ac02ce36"}
Nov 02 21:45:17.578082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.578061    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:17.578245 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.578099    4528 scope.go:122] "RemoveContainer" containerID="3f1b1104f67599d2a489389ad8128047c64afc1ee255719b6ab8a8b7ac02ce36"
Nov 02 21:45:17.578384 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:17.578361    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md_odh-ai-gateway-infra(a2763b03-c0d8-4c1d-8b46-8c6615749031)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031"
Nov 02 21:45:17.579840 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.579521    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:45:17.585335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.585318    4528 scope.go:122] "RemoveContainer" containerID="2e9ce0fc7945270244ed88f94ecadbefa351d5487de79f523e66cbfa8083712e"
Nov 02 21:45:17.598622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.598515    4528 scope.go:122] "RemoveContainer" containerID="143cdcb3e0bd6e80d9cb0ee59ef25724a6fa612ac09c6fccb95e41768cff3ea5"
Nov 02 21:45:17.624721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.624495    4528 scope.go:122] "RemoveContainer" containerID="05d0b04e2128e432f64f89e2ade53027e53bbab708c0eb276d71c84e1470db6f"
Nov 02 21:45:17.801346 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.801317    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"]
Nov 02 21:45:17.851800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.851772    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:17.981338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.981297    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987cde20-ab8f-4bce-a527-d57626938569-kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/987cde20-ab8f-4bce-a527-d57626938569-kube-api-access-p9chl\") pod \"987cde20-ab8f-4bce-a527-d57626938569\" (UID: \"987cde20-ab8f-4bce-a527-d57626938569\") "
Nov 02 21:45:17.981466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.981348    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987cde20-ab8f-4bce-a527-d57626938569-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cde20-ab8f-4bce-a527-d57626938569-maas-api-service-ca\") pod \"987cde20-ab8f-4bce-a527-d57626938569\" (UID: \"987cde20-ab8f-4bce-a527-d57626938569\") "
Nov 02 21:45:17.981698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.981676    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987cde20-ab8f-4bce-a527-d57626938569-maas-api-service-ca" pod "987cde20-ab8f-4bce-a527-d57626938569" (UID: "987cde20-ab8f-4bce-a527-d57626938569"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:17.983260 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:17.983239    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987cde20-ab8f-4bce-a527-d57626938569-kube-api-access-p9chl" pod "987cde20-ab8f-4bce-a527-d57626938569" (UID: "987cde20-ab8f-4bce-a527-d57626938569"). InnerVolumeSpecName "kube-api-access-p9chl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:18.021970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.021945    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"]
Nov 02 21:45:18.082688 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.082658    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/987cde20-ab8f-4bce-a527-d57626938569-kube-api-access-p9chl\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.082783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.082689    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987cde20-ab8f-4bce-a527-d57626938569-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.090983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.090949    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"]
Nov 02 21:45:18.100229 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.100204    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-5scx6"]
Nov 02 21:45:18.215365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.215340    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:45:18.285348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285318    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-envoy\") pod \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") "
Nov 02 21:45:18.285515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285372    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-socket\") pod \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") "
Nov 02 21:45:18.285515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285412    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-certs\") pod \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") "
Nov 02 21:45:18.285515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285485    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-token\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-token\") pod \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") "
Nov 02 21:45:18.285651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285554    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-podinfo\") pod \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") "
Nov 02 21:45:18.285651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285591    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istiod-ca-cert\") pod \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") "
Nov 02 21:45:18.285760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285674    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-credential-socket\") pod \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") "
Nov 02 21:45:18.285760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285679    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-socket" pod "e733fbf4-28fd-4a07-8357-a372bc92d8b3" (UID: "e733fbf4-28fd-4a07-8357-a372bc92d8b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:18.285760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285710    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-data\") pod \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") "
Nov 02 21:45:18.285912 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285765    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-kube-api-access-8cjxp\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-kube-api-access-8cjxp\") pod \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\" (UID: \"e733fbf4-28fd-4a07-8357-a372bc92d8b3\") "
Nov 02 21:45:18.285912 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.285763    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-certs" pod "e733fbf4-28fd-4a07-8357-a372bc92d8b3" (UID: "e733fbf4-28fd-4a07-8357-a372bc92d8b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:18.286092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.286065    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istiod-ca-cert" pod "e733fbf4-28fd-4a07-8357-a372bc92d8b3" (UID: "e733fbf4-28fd-4a07-8357-a372bc92d8b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:18.286721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.286293    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.286812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.286764    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.286812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.286501    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-credential-socket" pod "e733fbf4-28fd-4a07-8357-a372bc92d8b3" (UID: "e733fbf4-28fd-4a07-8357-a372bc92d8b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:18.286812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.286682    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-data" pod "e733fbf4-28fd-4a07-8357-a372bc92d8b3" (UID: "e733fbf4-28fd-4a07-8357-a372bc92d8b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:18.287671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.287648    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-envoy" pod "e733fbf4-28fd-4a07-8357-a372bc92d8b3" (UID: "e733fbf4-28fd-4a07-8357-a372bc92d8b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:18.288222 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.288198    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-podinfo" pod "e733fbf4-28fd-4a07-8357-a372bc92d8b3" (UID: "e733fbf4-28fd-4a07-8357-a372bc92d8b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:45:18.288304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.288235    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-token" pod "e733fbf4-28fd-4a07-8357-a372bc92d8b3" (UID: "e733fbf4-28fd-4a07-8357-a372bc92d8b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:18.288304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.288278    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-kube-api-access-8cjxp" pod "e733fbf4-28fd-4a07-8357-a372bc92d8b3" (UID: "e733fbf4-28fd-4a07-8357-a372bc92d8b3"). InnerVolumeSpecName "kube-api-access-8cjxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:18.357005 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.356970    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"]
Nov 02 21:45:18.359667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.359641    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894265-8zcq4"]
Nov 02 21:45:18.388137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.388096    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.388137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.388139    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.388297 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.388151    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.388297 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.388160    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.388297 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.388168    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.388297 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.388176    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cjxp\" (UniqueName: \"kubernetes.io/projected/e733fbf4-28fd-4a07-8357-a372bc92d8b3-kube-api-access-8cjxp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.388297 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.388185    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e733fbf4-28fd-4a07-8357-a372bc92d8b3-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.467458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.467432    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"]
Nov 02 21:45:18.470685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.470662    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894265-2vvxk"]
Nov 02 21:45:18.593784 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.593750    4528 generic.go:356] "Generic (PLEG): container finished" podID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerID="5464e91d6b9c81f8b6d6ecb68a49fe9ff1522ed486a50c0398d1581bb83ac1bc" exitCode=60
Nov 02 21:45:18.594220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.593819    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" event={"ID":"bec2d0e4-4b05-4e38-8865-d30dd5085528","Type":"ContainerDied","Data":"5464e91d6b9c81f8b6d6ecb68a49fe9ff1522ed486a50c0398d1581bb83ac1bc"}
Nov 02 21:45:18.594220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.593886    4528 scope.go:122] "RemoveContainer" containerID="26de1ef5926c80a65727e8ff00e305bc58938885a11d5ed4cf1cfd3011d289f5"
Nov 02 21:45:18.594359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.594339    4528 scope.go:122] "RemoveContainer" containerID="5464e91d6b9c81f8b6d6ecb68a49fe9ff1522ed486a50c0398d1581bb83ac1bc"
Nov 02 21:45:18.594620 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:18.594599    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4_odh-ai-gateway-infra(bec2d0e4-4b05-4e38-8865-d30dd5085528)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528"
Nov 02 21:45:18.599602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.599576    4528 generic.go:356] "Generic (PLEG): container finished" podID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerID="8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d" exitCode=0
Nov 02 21:45:18.599715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.599660    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" event={"ID":"e733fbf4-28fd-4a07-8357-a372bc92d8b3","Type":"ContainerDied","Data":"8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d"}
Nov 02 21:45:18.599775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.599711    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk" event={"ID":"e733fbf4-28fd-4a07-8357-a372bc92d8b3","Type":"ContainerDied","Data":"6d420d7a161add09f2aa9bcbf875ba7485ed142f4e7d731279b6e91de9123c4c"}
Nov 02 21:45:18.599775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.599753    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"
Nov 02 21:45:18.602800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.602777    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"
Nov 02 21:45:18.614183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.614155    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4" event={"ID":"987cde20-ab8f-4bce-a527-d57626938569","Type":"ContainerDied","Data":"9a69ee0d3be38818c50f9dd913dfb665e3f5cf801bae9ba73373b96c28051be9"}
Nov 02 21:45:18.618814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.618790    4528 scope.go:122] "RemoveContainer" containerID="8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d"
Nov 02 21:45:18.643869 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.643845    4528 scope.go:122] "RemoveContainer" containerID="6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f"
Nov 02 21:45:18.661561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.661543    4528 scope.go:122] "RemoveContainer" containerID="8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d"
Nov 02 21:45:18.662442 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:18.662075    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d\": container with ID starting with 8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d not found: ID does not exist" containerID="8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d"
Nov 02 21:45:18.662821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.662583    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d"} err="failed to get container status \"8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d\": rpc error: code = NotFound desc = could not find container \"8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d\": container with ID starting with 8692135d090be3ad1fa0b83fa3c1e80ead0caaaaa9970f9635a7e897e9f5b95d not found: ID does not exist"
Nov 02 21:45:18.662821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.662616    4528 scope.go:122] "RemoveContainer" containerID="6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f"
Nov 02 21:45:18.663320 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:18.663058    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f\": container with ID starting with 6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f not found: ID does not exist" containerID="6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f"
Nov 02 21:45:18.663320 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.663083    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f"} err="failed to get container status \"6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f\": rpc error: code = NotFound desc = could not find container \"6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f\": container with ID starting with 6a9897a29b7bb56d8e73e423bbc38b3d3f5bfd884255da8055deab0177bef39f not found: ID does not exist"
Nov 02 21:45:18.663320 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.663098    4528 scope.go:122] "RemoveContainer" containerID="711d2cff467be22749e3fb8cc81bc7028ba83397e525c11307db7a0ebdd514b6"
Nov 02 21:45:18.744581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.744556    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:18.788625 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.788603    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:18.894082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.894010    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8e4c51-545b-4043-8bbc-6314c1e91839-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8e4c51-545b-4043-8bbc-6314c1e91839-maas-api-service-ca\") pod \"2f8e4c51-545b-4043-8bbc-6314c1e91839\" (UID: \"2f8e4c51-545b-4043-8bbc-6314c1e91839\") "
Nov 02 21:45:18.894242 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.894143    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2763b03-c0d8-4c1d-8b46-8c6615749031-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2763b03-c0d8-4c1d-8b46-8c6615749031-maas-api-service-ca\") pod \"a2763b03-c0d8-4c1d-8b46-8c6615749031\" (UID: \"a2763b03-c0d8-4c1d-8b46-8c6615749031\") "
Nov 02 21:45:18.894284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.894240    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8e4c51-545b-4043-8bbc-6314c1e91839-kube-api-access-wclhw\" (UniqueName: \"kubernetes.io/projected/2f8e4c51-545b-4043-8bbc-6314c1e91839-kube-api-access-wclhw\") pod \"2f8e4c51-545b-4043-8bbc-6314c1e91839\" (UID: \"2f8e4c51-545b-4043-8bbc-6314c1e91839\") "
Nov 02 21:45:18.894284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.894269    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2763b03-c0d8-4c1d-8b46-8c6615749031-kube-api-access-9p9wj\" (UniqueName: \"kubernetes.io/projected/a2763b03-c0d8-4c1d-8b46-8c6615749031-kube-api-access-9p9wj\") pod \"a2763b03-c0d8-4c1d-8b46-8c6615749031\" (UID: \"a2763b03-c0d8-4c1d-8b46-8c6615749031\") "
Nov 02 21:45:18.894442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.894423    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8e4c51-545b-4043-8bbc-6314c1e91839-maas-api-service-ca" pod "2f8e4c51-545b-4043-8bbc-6314c1e91839" (UID: "2f8e4c51-545b-4043-8bbc-6314c1e91839"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:18.894481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.894438    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2763b03-c0d8-4c1d-8b46-8c6615749031-maas-api-service-ca" pod "a2763b03-c0d8-4c1d-8b46-8c6615749031" (UID: "a2763b03-c0d8-4c1d-8b46-8c6615749031"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:18.894761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.894738    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2763b03-c0d8-4c1d-8b46-8c6615749031-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.894761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.894762    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8e4c51-545b-4043-8bbc-6314c1e91839-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.896267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.896247    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8e4c51-545b-4043-8bbc-6314c1e91839-kube-api-access-wclhw" pod "2f8e4c51-545b-4043-8bbc-6314c1e91839" (UID: "2f8e4c51-545b-4043-8bbc-6314c1e91839"). InnerVolumeSpecName "kube-api-access-wclhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:18.896343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.896267    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2763b03-c0d8-4c1d-8b46-8c6615749031-kube-api-access-9p9wj" pod "a2763b03-c0d8-4c1d-8b46-8c6615749031" (UID: "a2763b03-c0d8-4c1d-8b46-8c6615749031"). InnerVolumeSpecName "kube-api-access-9p9wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:18.913553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.913520    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" path="/var/lib/kubelet/pods/453701cb-6a3f-48ae-8cb0-6831a58e06ef/volumes"
Nov 02 21:45:18.913977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.913958    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" path="/var/lib/kubelet/pods/99712198-d097-4fed-ba4a-76bfa3888fee/volumes"
Nov 02 21:45:18.914421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.914401    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" path="/var/lib/kubelet/pods/f3b328bc-d9e2-4821-8e0e-3366a1d84799/volumes"
Nov 02 21:45:18.995908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.995882    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wclhw\" (UniqueName: \"kubernetes.io/projected/2f8e4c51-545b-4043-8bbc-6314c1e91839-kube-api-access-wclhw\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.995908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.995908    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p9wj\" (UniqueName: \"kubernetes.io/projected/a2763b03-c0d8-4c1d-8b46-8c6615749031-kube-api-access-9p9wj\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:18.996350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.996331    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"]
Nov 02 21:45:18.998940 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:18.998918    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a9abc-openshift-default-79c549bc89-xr4dk"]
Nov 02 21:45:19.168826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.168754    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"]
Nov 02 21:45:19.171964 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.171942    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894265-s55t4"]
Nov 02 21:45:19.610712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.610684    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"
Nov 02 21:45:19.611188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.610684    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255" event={"ID":"2f8e4c51-545b-4043-8bbc-6314c1e91839","Type":"ContainerDied","Data":"5b7171e4dd405880a03050c878fd95570a089f47447e9d21c2bde26a5431a1bd"}
Nov 02 21:45:19.611188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.610816    4528 scope.go:122] "RemoveContainer" containerID="2cc569d8aa33cbb2624dff5a85a9f25b349aa6bb510024a80b2d570fe82ee7c7"
Nov 02 21:45:19.615218 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.615196    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"
Nov 02 21:45:19.615348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.615221    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md" event={"ID":"a2763b03-c0d8-4c1d-8b46-8c6615749031","Type":"ContainerDied","Data":"5d0429b4dbb1accf576c0856bf26af3a1e88655fdbc2349042eb870c4a63fe9e"}
Nov 02 21:45:19.621086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.621063    4528 scope.go:122] "RemoveContainer" containerID="3f1b1104f67599d2a489389ad8128047c64afc1ee255719b6ab8a8b7ac02ce36"
Nov 02 21:45:19.652571 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.652548    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:45:19.652660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.652598    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:45:19.708820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.708799    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"]
Nov 02 21:45:19.781579 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.781543    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"]
Nov 02 21:45:19.781705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.781606    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e12f99-29894265-mh255"]
Nov 02 21:45:19.844829 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.844807    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:19.885682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.885624    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"]
Nov 02 21:45:19.888048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.888031    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3e3997-29894265-k54md"]
Nov 02 21:45:19.906841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.906822    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec2d0e4-4b05-4e38-8865-d30dd5085528-kube-api-access-s9px7\" (UniqueName: \"kubernetes.io/projected/bec2d0e4-4b05-4e38-8865-d30dd5085528-kube-api-access-s9px7\") pod \"bec2d0e4-4b05-4e38-8865-d30dd5085528\" (UID: \"bec2d0e4-4b05-4e38-8865-d30dd5085528\") "
Nov 02 21:45:19.907039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.907021    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec2d0e4-4b05-4e38-8865-d30dd5085528-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d0e4-4b05-4e38-8865-d30dd5085528-maas-api-service-ca\") pod \"bec2d0e4-4b05-4e38-8865-d30dd5085528\" (UID: \"bec2d0e4-4b05-4e38-8865-d30dd5085528\") "
Nov 02 21:45:19.907461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.907431    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec2d0e4-4b05-4e38-8865-d30dd5085528-maas-api-service-ca" pod "bec2d0e4-4b05-4e38-8865-d30dd5085528" (UID: "bec2d0e4-4b05-4e38-8865-d30dd5085528"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:19.908868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:19.908844    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec2d0e4-4b05-4e38-8865-d30dd5085528-kube-api-access-s9px7" pod "bec2d0e4-4b05-4e38-8865-d30dd5085528" (UID: "bec2d0e4-4b05-4e38-8865-d30dd5085528"). InnerVolumeSpecName "kube-api-access-s9px7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:20.009050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.009010    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec2d0e4-4b05-4e38-8865-d30dd5085528-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:20.009050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.009041    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9px7\" (UniqueName: \"kubernetes.io/projected/bec2d0e4-4b05-4e38-8865-d30dd5085528-kube-api-access-s9px7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:20.047506 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.047479    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"]
Nov 02 21:45:20.049953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.049933    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cab2a9-tchsz"]
Nov 02 21:45:20.632063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.632030    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"
Nov 02 21:45:20.632063 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.632052    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4" event={"ID":"bec2d0e4-4b05-4e38-8865-d30dd5085528","Type":"ContainerDied","Data":"5898e007aad9d6bc95a74fd832be34d5669550c4981655fe72f366ca644b8713"}
Nov 02 21:45:20.632626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.632178    4528 scope.go:122] "RemoveContainer" containerID="5464e91d6b9c81f8b6d6ecb68a49fe9ff1522ed486a50c0398d1581bb83ac1bc"
Nov 02 21:45:20.796857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.796829    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"]
Nov 02 21:45:20.801334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.801309    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ce2453-29894265-j6xw4"]
Nov 02 21:45:20.913677 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.913602    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" path="/var/lib/kubelet/pods/2f8e4c51-545b-4043-8bbc-6314c1e91839/volumes"
Nov 02 21:45:20.914019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.914006    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987cde20-ab8f-4bce-a527-d57626938569" path="/var/lib/kubelet/pods/987cde20-ab8f-4bce-a527-d57626938569/volumes"
Nov 02 21:45:20.914388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.914375    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" path="/var/lib/kubelet/pods/a2763b03-c0d8-4c1d-8b46-8c6615749031/volumes"
Nov 02 21:45:20.915042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.915028    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc30884-92a0-484e-a7a6-be87b920e81b" path="/var/lib/kubelet/pods/acc30884-92a0-484e-a7a6-be87b920e81b/volumes"
Nov 02 21:45:20.915396 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.915383    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" path="/var/lib/kubelet/pods/bec2d0e4-4b05-4e38-8865-d30dd5085528/volumes"
Nov 02 21:45:20.915775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:20.915763    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" path="/var/lib/kubelet/pods/e733fbf4-28fd-4a07-8357-a372bc92d8b3/volumes"
Nov 02 21:45:22.648095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.648066    4528 generic.go:356] "Generic (PLEG): container finished" podID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerID="102c0939bd138096ad2b395c599010edb1c5aacab26a9d7f59a7ccef12e2d613" exitCode=0
Nov 02 21:45:22.648449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.648157    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" event={"ID":"7a206f69-8091-412f-9a80-e89cd60e50a5","Type":"ContainerDied","Data":"102c0939bd138096ad2b395c599010edb1c5aacab26a9d7f59a7ccef12e2d613"}
Nov 02 21:45:22.648449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.648236    4528 scope.go:122] "RemoveContainer" containerID="75d747d1bf64e973afbb06870b021192b3ea0ae365b6b9f41fa18ef040a1a4b0"
Nov 02 21:45:22.784753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.784723    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:45:22.944775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.944687    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-podinfo\") pod \"7a206f69-8091-412f-9a80-e89cd60e50a5\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") "
Nov 02 21:45:22.944919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.944790    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-certs\") pod \"7a206f69-8091-412f-9a80-e89cd60e50a5\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") "
Nov 02 21:45:22.944919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.944886    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-socket\") pod \"7a206f69-8091-412f-9a80-e89cd60e50a5\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") "
Nov 02 21:45:22.944919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.944908    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-envoy\") pod \"7a206f69-8091-412f-9a80-e89cd60e50a5\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") "
Nov 02 21:45:22.945065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.944933    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-credential-socket\") pod \"7a206f69-8091-412f-9a80-e89cd60e50a5\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") "
Nov 02 21:45:22.945065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.944972    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-kube-api-access-pqrmm\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-kube-api-access-pqrmm\") pod \"7a206f69-8091-412f-9a80-e89cd60e50a5\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") "
Nov 02 21:45:22.945065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945002    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-token\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-token\") pod \"7a206f69-8091-412f-9a80-e89cd60e50a5\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") "
Nov 02 21:45:22.945065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945028    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a206f69-8091-412f-9a80-e89cd60e50a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206f69-8091-412f-9a80-e89cd60e50a5-istiod-ca-cert\") pod \"7a206f69-8091-412f-9a80-e89cd60e50a5\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") "
Nov 02 21:45:22.945065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945047    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-data\") pod \"7a206f69-8091-412f-9a80-e89cd60e50a5\" (UID: \"7a206f69-8091-412f-9a80-e89cd60e50a5\") "
Nov 02 21:45:22.945349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945102    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-certs" pod "7a206f69-8091-412f-9a80-e89cd60e50a5" (UID: "7a206f69-8091-412f-9a80-e89cd60e50a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:22.945349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945176    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-credential-socket" pod "7a206f69-8091-412f-9a80-e89cd60e50a5" (UID: "7a206f69-8091-412f-9a80-e89cd60e50a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:22.945349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945189    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-socket" pod "7a206f69-8091-412f-9a80-e89cd60e50a5" (UID: "7a206f69-8091-412f-9a80-e89cd60e50a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:22.945543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945382    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-data" pod "7a206f69-8091-412f-9a80-e89cd60e50a5" (UID: "7a206f69-8091-412f-9a80-e89cd60e50a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:22.945543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945441    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a206f69-8091-412f-9a80-e89cd60e50a5-istiod-ca-cert" pod "7a206f69-8091-412f-9a80-e89cd60e50a5" (UID: "7a206f69-8091-412f-9a80-e89cd60e50a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:22.945651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945605    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:22.945651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945626    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:22.945651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945642    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a206f69-8091-412f-9a80-e89cd60e50a5-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:22.945795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945654    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:22.945795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.945668    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:22.946899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.946878    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-podinfo" pod "7a206f69-8091-412f-9a80-e89cd60e50a5" (UID: "7a206f69-8091-412f-9a80-e89cd60e50a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:45:22.947147 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.947097    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-envoy" pod "7a206f69-8091-412f-9a80-e89cd60e50a5" (UID: "7a206f69-8091-412f-9a80-e89cd60e50a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:45:22.947257 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.947238    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-kube-api-access-pqrmm" pod "7a206f69-8091-412f-9a80-e89cd60e50a5" (UID: "7a206f69-8091-412f-9a80-e89cd60e50a5"). InnerVolumeSpecName "kube-api-access-pqrmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:22.947307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:22.947265    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-token" pod "7a206f69-8091-412f-9a80-e89cd60e50a5" (UID: "7a206f69-8091-412f-9a80-e89cd60e50a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:23.046521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.046493    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:23.046521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.046519    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrmm\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-kube-api-access-pqrmm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:23.046711 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.046528    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:23.046711 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.046536    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a206f69-8091-412f-9a80-e89cd60e50a5-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:23.611628 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.611593    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"]
Nov 02 21:45:23.612654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.612629    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" containerName="cleanup"
Nov 02 21:45:23.612821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.612807    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" containerName="cleanup"
Nov 02 21:45:23.612917 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.612909    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerName="cleanup"
Nov 02 21:45:23.612999 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.612990    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerName="cleanup"
Nov 02 21:45:23.613057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613048    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerName="cleanup"
Nov 02 21:45:23.613103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613096    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerName="cleanup"
Nov 02 21:45:23.613179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613172    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerName="cleanup"
Nov 02 21:45:23.613236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613227    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerName="cleanup"
Nov 02 21:45:23.613329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613321    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerName="cleanup"
Nov 02 21:45:23.613394 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613385    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerName="cleanup"
Nov 02 21:45:23.613471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613462    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy"
Nov 02 21:45:23.613534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613525    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy"
Nov 02 21:45:23.613597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613589    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy"
Nov 02 21:45:23.613655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613646    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy"
Nov 02 21:45:23.613726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613718    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerName="cleanup"
Nov 02 21:45:23.613784 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613776    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerName="cleanup"
Nov 02 21:45:23.613852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613844    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy"
Nov 02 21:45:23.613912 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613904    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy"
Nov 02 21:45:23.613975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.613967    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy"
Nov 02 21:45:23.614034 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614024    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy"
Nov 02 21:45:23.614121 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614099    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerName="cleanup"
Nov 02 21:45:23.614225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614202    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerName="cleanup"
Nov 02 21:45:23.614225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614222    4528 cpu_manager.go:414] "containerMap: removing container" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614229    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614241    4528 cpu_manager.go:414] "containerMap: removing container" podUID="987cde20-ab8f-4bce-a527-d57626938569" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614250    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987cde20-ab8f-4bce-a527-d57626938569" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614259    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614266    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614278    4528 cpu_manager.go:414] "containerMap: removing container" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614285    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614294    4528 cpu_manager.go:414] "containerMap: removing container" podUID="987cde20-ab8f-4bce-a527-d57626938569" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614302    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987cde20-ab8f-4bce-a527-d57626938569" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614314    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerName="cleanup"
Nov 02 21:45:23.614332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614321    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614343    4528 cpu_manager.go:414] "containerMap: removing container" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614350    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614361    4528 cpu_manager.go:414] "containerMap: removing container" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614368    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614386    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614392    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614408    4528 cpu_manager.go:414] "containerMap: removing container" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614415    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614425    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614432    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614563    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614576    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614588    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614598    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614607    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" containerName="istio-proxy"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614616    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e733fbf4-28fd-4a07-8357-a372bc92d8b3" containerName="istio-proxy"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614627    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614639    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614646    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614655    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="987cde20-ab8f-4bce-a527-d57626938569" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614664    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614674    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614682    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="987cde20-ab8f-4bce-a527-d57626938569" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614693    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614702    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614711    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614722    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="99712198-d097-4fed-ba4a-76bfa3888fee" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614733    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614742    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad43ab3f-245e-4af2-9b9e-0de944ccc0ce" containerName="cleanup"
Nov 02 21:45:23.614854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.614753    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerName="cleanup"
Nov 02 21:45:23.619078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.619053    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:23.657698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.657664    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"
Nov 02 21:45:23.658093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.657703    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2" event={"ID":"7a206f69-8091-412f-9a80-e89cd60e50a5","Type":"ContainerDied","Data":"1117088584f071a570ce729ed6b701110f5e81b57ecbe1e146975e8cbbdd5dab"}
Nov 02 21:45:23.658093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.657758    4528 scope.go:122] "RemoveContainer" containerID="102c0939bd138096ad2b395c599010edb1c5aacab26a9d7f59a7ccef12e2d613"
Nov 02 21:45:23.665026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.665002    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"]
Nov 02 21:45:23.672533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.672508    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-8e90da\""
Nov 02 21:45:23.672646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.672518    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-8e90da-dockercfg-tfkn5\""
Nov 02 21:45:23.769131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.760855    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6dr\" (UniqueName: \"kubernetes.io/projected/6644f753-dd82-4c10-8775-72ce9e609997-kube-api-access-fq6dr\") pod \"payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k\" (UID: \"6644f753-dd82-4c10-8775-72ce9e609997\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:23.769131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.761071    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644f753-dd82-4c10-8775-72ce9e609997-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k\" (UID: \"6644f753-dd82-4c10-8775-72ce9e609997\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:23.773857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.773828    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"]
Nov 02 21:45:23.775338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.775310    4528 cpu_manager.go:414] "containerMap: removing container" podUID="987cde20-ab8f-4bce-a527-d57626938569" containerName="cleanup"
Nov 02 21:45:23.775514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.775341    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987cde20-ab8f-4bce-a527-d57626938569" containerName="cleanup"
Nov 02 21:45:23.775514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.775389    4528 cpu_manager.go:414] "containerMap: removing container" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerName="cleanup"
Nov 02 21:45:23.775514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.775397    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerName="cleanup"
Nov 02 21:45:23.775514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.775408    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerName="cleanup"
Nov 02 21:45:23.775514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.775415    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerName="cleanup"
Nov 02 21:45:23.775514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.775469    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerName="cleanup"
Nov 02 21:45:23.775514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.775481    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerName="cleanup"
Nov 02 21:45:23.777251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.776333    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8e4c51-545b-4043-8bbc-6314c1e91839" containerName="cleanup"
Nov 02 21:45:23.777251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.776396    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec2d0e4-4b05-4e38-8865-d30dd5085528" containerName="cleanup"
Nov 02 21:45:23.777251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.776408    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2763b03-c0d8-4c1d-8b46-8c6615749031" containerName="cleanup"
Nov 02 21:45:23.777251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.776422    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="987cde20-ab8f-4bce-a527-d57626938569" containerName="cleanup"
Nov 02 21:45:23.777251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.776431    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b328bc-d9e2-4821-8e0e-3366a1d84799" containerName="cleanup"
Nov 02 21:45:23.777251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.776441    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="453701cb-6a3f-48ae-8cb0-6831a58e06ef" containerName="cleanup"
Nov 02 21:45:23.783955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.783590    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:23.819975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.819941    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"]
Nov 02 21:45:23.861862 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.861779    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b970845-70c6-47e3-a1d6-f5d2527877f1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x\" (UID: \"9b970845-70c6-47e3-a1d6-f5d2527877f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:23.862010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.861962    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644f753-dd82-4c10-8775-72ce9e609997-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k\" (UID: \"6644f753-dd82-4c10-8775-72ce9e609997\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:23.862073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.862058    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6dr\" (UniqueName: \"kubernetes.io/projected/6644f753-dd82-4c10-8775-72ce9e609997-kube-api-access-fq6dr\") pod \"payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k\" (UID: \"6644f753-dd82-4c10-8775-72ce9e609997\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:23.862157 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.862138    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc25c\" (UniqueName: \"kubernetes.io/projected/9b970845-70c6-47e3-a1d6-f5d2527877f1-kube-api-access-mc25c\") pod \"payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x\" (UID: \"9b970845-70c6-47e3-a1d6-f5d2527877f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:23.862634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.862615    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644f753-dd82-4c10-8775-72ce9e609997-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k\" (UID: \"6644f753-dd82-4c10-8775-72ce9e609997\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:23.866685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.866660    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"]
Nov 02 21:45:23.871487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.871467    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7d3617-openshift-default-68bbbcb57f-x7nl2"]
Nov 02 21:45:23.918864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.918835    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6dr\" (UniqueName: \"kubernetes.io/projected/6644f753-dd82-4c10-8775-72ce9e609997-kube-api-access-fq6dr\") pod \"payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k\" (UID: \"6644f753-dd82-4c10-8775-72ce9e609997\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:23.935379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.935360    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:23.962809 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.962782    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc25c\" (UniqueName: \"kubernetes.io/projected/9b970845-70c6-47e3-a1d6-f5d2527877f1-kube-api-access-mc25c\") pod \"payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x\" (UID: \"9b970845-70c6-47e3-a1d6-f5d2527877f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:23.962911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.962831    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b970845-70c6-47e3-a1d6-f5d2527877f1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x\" (UID: \"9b970845-70c6-47e3-a1d6-f5d2527877f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:23.963551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:23.963527    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b970845-70c6-47e3-a1d6-f5d2527877f1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x\" (UID: \"9b970845-70c6-47e3-a1d6-f5d2527877f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:24.020280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.020252    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc25c\" (UniqueName: \"kubernetes.io/projected/9b970845-70c6-47e3-a1d6-f5d2527877f1-kube-api-access-mc25c\") pod \"payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x\" (UID: \"9b970845-70c6-47e3-a1d6-f5d2527877f1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:24.103263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.103239    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:24.324856 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:24.324824    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6644f753_dd82_4c10_8775_72ce9e609997.slice/crio-5f385c153eadf7dd1c95ccc06e876b724d3032bbd06c498e61c90c289570959d WatchSource:0}: Error finding container 5f385c153eadf7dd1c95ccc06e876b724d3032bbd06c498e61c90c289570959d: Status 404 returned error can't find the container with id 5f385c153eadf7dd1c95ccc06e876b724d3032bbd06c498e61c90c289570959d
Nov 02 21:45:24.327803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.327777    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"]
Nov 02 21:45:24.474519 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:24.474480    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b970845_70c6_47e3_a1d6_f5d2527877f1.slice/crio-1755fec8c3cb4618b39362c510feb5799688fdb8377718eaf2fd74d39730f007 WatchSource:0}: Error finding container 1755fec8c3cb4618b39362c510feb5799688fdb8377718eaf2fd74d39730f007: Status 404 returned error can't find the container with id 1755fec8c3cb4618b39362c510feb5799688fdb8377718eaf2fd74d39730f007
Nov 02 21:45:24.477922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.477901    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"]
Nov 02 21:45:24.664998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.664908    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x" event={"ID":"9b970845-70c6-47e3-a1d6-f5d2527877f1","Type":"ContainerStarted","Data":"e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226"}
Nov 02 21:45:24.664998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.664968    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x" event={"ID":"9b970845-70c6-47e3-a1d6-f5d2527877f1","Type":"ContainerStarted","Data":"1755fec8c3cb4618b39362c510feb5799688fdb8377718eaf2fd74d39730f007"}
Nov 02 21:45:24.664998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.664995    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:24.667455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.667427    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k" event={"ID":"6644f753-dd82-4c10-8775-72ce9e609997","Type":"ContainerStarted","Data":"e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78"}
Nov 02 21:45:24.667551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.667468    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k" event={"ID":"6644f753-dd82-4c10-8775-72ce9e609997","Type":"ContainerStarted","Data":"5f385c153eadf7dd1c95ccc06e876b724d3032bbd06c498e61c90c289570959d"}
Nov 02 21:45:24.667551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.667488    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:24.780173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.780099    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x" podStartSLOduration=1.780086967 podStartE2EDuration="1.780086967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:24.777842383 +0000 UTC m=+2212.041378700" watchObservedRunningTime="2026-11-02 21:45:24.780086967 +0000 UTC m=+2212.043623283"
Nov 02 21:45:24.908080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.899658    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k" podStartSLOduration=1.899611374 podStartE2EDuration="1.899611374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:24.890861079 +0000 UTC m=+2212.154397420" watchObservedRunningTime="2026-11-02 21:45:24.899611374 +0000 UTC m=+2212.163147692"
Nov 02 21:45:24.936086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:24.935617    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a206f69-8091-412f-9a80-e89cd60e50a5" path="/var/lib/kubelet/pods/7a206f69-8091-412f-9a80-e89cd60e50a5/volumes"
Nov 02 21:45:27.323886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:27.323846    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:45:34.637803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:34.637773    4528 scope.go:122] "RemoveContainer" containerID="ac685e660fb0d21325b3fe53178babd3f737698c90121c8b19e78d5df99e0c89"
Nov 02 21:45:35.677515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:35.677486    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:45:35.677893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:35.677568    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:45:37.652336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:37.652303    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"]
Nov 02 21:45:37.652825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:37.652547    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q" podUID="d91cdb17-b6df-46f8-b1cf-99287b1d68a7" containerName="payload-processing" containerID="cri-o://b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64" gracePeriod=30
Nov 02 21:45:37.680884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:37.680851    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"]
Nov 02 21:45:37.681089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:37.681067    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b" podUID="c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2" containerName="payload-pre-processing" containerID="cri-o://2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b" gracePeriod=30
Nov 02 21:45:37.944551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:37.944519    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:45:37.969065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:37.969044    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:45:38.148886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.139082    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-plugins-config-volume\") pod \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\" (UID: \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\") "
Nov 02 21:45:38.148886 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.139268    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-kube-api-access-cbs6j\" (UniqueName: \"kubernetes.io/projected/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-kube-api-access-cbs6j\") pod \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\" (UID: \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\") "
Nov 02 21:45:38.150345 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.149323    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-plugins-config-volume\") pod \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\" (UID: \"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2\") "
Nov 02 21:45:38.150345 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.149429    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-kube-api-access-nl64z\" (UniqueName: \"kubernetes.io/projected/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-kube-api-access-nl64z\") pod \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\" (UID: \"d91cdb17-b6df-46f8-b1cf-99287b1d68a7\") "
Nov 02 21:45:38.152334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.152308    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-plugins-config-volume" pod "c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2" (UID: "c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:38.152860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.152839    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-plugins-config-volume" pod "d91cdb17-b6df-46f8-b1cf-99287b1d68a7" (UID: "d91cdb17-b6df-46f8-b1cf-99287b1d68a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:45:38.155564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.155531    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-kube-api-access-cbs6j" pod "c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2" (UID: "c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2"). InnerVolumeSpecName "kube-api-access-cbs6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:38.185379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.185270    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-kube-api-access-nl64z" pod "d91cdb17-b6df-46f8-b1cf-99287b1d68a7" (UID: "d91cdb17-b6df-46f8-b1cf-99287b1d68a7"). InnerVolumeSpecName "kube-api-access-nl64z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:45:38.250915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.250879    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:38.250915 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.250916    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbs6j\" (UniqueName: \"kubernetes.io/projected/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-kube-api-access-cbs6j\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:38.251148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.250931    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:38.251148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.250945    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl64z\" (UniqueName: \"kubernetes.io/projected/d91cdb17-b6df-46f8-b1cf-99287b1d68a7-kube-api-access-nl64z\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:45:38.752734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.752698    4528 generic.go:356] "Generic (PLEG): container finished" podID="c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2" containerID="2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b" exitCode=0
Nov 02 21:45:38.753191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.752747    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"
Nov 02 21:45:38.753191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.752794    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b" event={"ID":"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2","Type":"ContainerDied","Data":"2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b"}
Nov 02 21:45:38.753191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.752849    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b" event={"ID":"c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2","Type":"ContainerDied","Data":"fd2b469abb3d27040ae72a019eb0a0643d1682b9ca328b8aa71e62f778c68227"}
Nov 02 21:45:38.753191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.752875    4528 scope.go:122] "RemoveContainer" containerID="2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b"
Nov 02 21:45:38.754404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.754385    4528 generic.go:356] "Generic (PLEG): container finished" podID="d91cdb17-b6df-46f8-b1cf-99287b1d68a7" containerID="b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64" exitCode=0
Nov 02 21:45:38.754516 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.754432    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"
Nov 02 21:45:38.754516 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.754446    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q" event={"ID":"d91cdb17-b6df-46f8-b1cf-99287b1d68a7","Type":"ContainerDied","Data":"b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64"}
Nov 02 21:45:38.754516 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.754477    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q" event={"ID":"d91cdb17-b6df-46f8-b1cf-99287b1d68a7","Type":"ContainerDied","Data":"92b53c4048a6ccbdf9924fdde1faf62fd2b77fce792ebb401653c8aee27514a2"}
Nov 02 21:45:38.765558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.765537    4528 scope.go:122] "RemoveContainer" containerID="2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b"
Nov 02 21:45:38.765966 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:38.765849    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b\": container with ID starting with 2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b not found: ID does not exist" containerID="2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b"
Nov 02 21:45:38.765966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.765872    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b"} err="failed to get container status \"2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b\": rpc error: code = NotFound desc = could not find container \"2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b\": container with ID starting with 2b95543effddc68a54cd6d5c961509901ea22158710f86a4bf251d2b6ed7c85b not found: ID does not exist"
Nov 02 21:45:38.765966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.765892    4528 scope.go:122] "RemoveContainer" containerID="b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64"
Nov 02 21:45:38.774826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.774812    4528 scope.go:122] "RemoveContainer" containerID="b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64"
Nov 02 21:45:38.775075 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:38.775056    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64\": container with ID starting with b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64 not found: ID does not exist" containerID="b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64"
Nov 02 21:45:38.775155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.775081    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64"} err="failed to get container status \"b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64\": rpc error: code = NotFound desc = could not find container \"b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64\": container with ID starting with b0bd4accfd5ee489869e39b149849993388e547fa49fb6dc355f616891069e64 not found: ID does not exist"
Nov 02 21:45:38.917050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.917020    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"]
Nov 02 21:45:38.967208 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:38.967135    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7d756b94f9-x7q9b"]
Nov 02 21:45:39.075470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:39.075435    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"]
Nov 02 21:45:39.078313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:39.078288    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-387f88-5b584445b4-g696q"]
Nov 02 21:45:40.913038 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:40.913006    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2" path="/var/lib/kubelet/pods/c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2/volumes"
Nov 02 21:45:40.913433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:40.913396    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91cdb17-b6df-46f8-b1cf-99287b1d68a7" path="/var/lib/kubelet/pods/d91cdb17-b6df-46f8-b1cf-99287b1d68a7/volumes"
Nov 02 21:45:44.112946 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:45:44.112903    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:45:51.319599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.319568    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"]
Nov 02 21:45:51.320231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.320211    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d91cdb17-b6df-46f8-b1cf-99287b1d68a7" containerName="payload-processing"
Nov 02 21:45:51.320336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.320233    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91cdb17-b6df-46f8-b1cf-99287b1d68a7" containerName="payload-processing"
Nov 02 21:45:51.320336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.320252    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2" containerName="payload-pre-processing"
Nov 02 21:45:51.320336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.320257    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2" containerName="payload-pre-processing"
Nov 02 21:45:51.320467 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.320385    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91cdb17-b6df-46f8-b1cf-99287b1d68a7" containerName="payload-processing"
Nov 02 21:45:51.320467 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.320420    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72f5c1b-6201-4c2d-a8aa-8d17bd7169e2" containerName="payload-pre-processing"
Nov 02 21:45:51.325276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.325252    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.399058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.399017    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-data\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.408304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.408266    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-credential-socket\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.408546 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.408526    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-certs\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.421096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.408781    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-socket\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.421393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.421360    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-envoy\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.425755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.425673    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-podinfo\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.425998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.425958    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35d6134-28bb-4351-943d-0bd7d643dbe9-istiod-ca-cert\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.426432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.426238    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-token\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.426931 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.426905    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs6jg\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-kube-api-access-fs6jg\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.437230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.433800    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"]
Nov 02 21:45:51.476848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.476815    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cc0d6a-openshift-default-dockercfg-lk6ts\""
Nov 02 21:45:51.528020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.527990    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-token\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528031    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs6jg\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-kube-api-access-fs6jg\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528152    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-data\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528227    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-credential-socket\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528250    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-certs\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528340    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-socket\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528416 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528383    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-envoy\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528428    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-podinfo\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528456    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35d6134-28bb-4351-943d-0bd7d643dbe9-istiod-ca-cert\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528597    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-data\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528635    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-credential-socket\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528677    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-certs\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.528851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.528801    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-socket\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.529219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.529197    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35d6134-28bb-4351-943d-0bd7d643dbe9-istiod-ca-cert\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.530670 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.530643    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-envoy\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.530832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.530815    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-podinfo\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.585138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.585050    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-token\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.585373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.585352    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs6jg\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-kube-api-access-fs6jg\") pod \"e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") " pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.640783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.640760    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:51.658868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.658842    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"]
Nov 02 21:45:51.664722 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.664703    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.718101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.718051    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-aa8aff-openshift-default-dockercfg-dwvhh\""
Nov 02 21:45:51.719817 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.719787    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"]
Nov 02 21:45:51.840530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.835451    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-podinfo\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.840530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.835655    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-credential-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.840530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.835691    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpg6w\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-kube-api-access-rpg6w\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.840530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.835874    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istiod-ca-cert\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.840530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.835962    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-data\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.840530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.836017    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-token\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.840530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.836058    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-certs\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.840530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.836087    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-envoy\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.840530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.836134    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.936991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.936953    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-data\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.937227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.937019    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-token\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.937227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.937062    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-certs\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.937227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.937087    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-envoy\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.937227 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.937130    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.937541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.937515    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.937691 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.937622    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-podinfo\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.937753 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.937723    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-credential-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.937791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.937749    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpg6w\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-kube-api-access-rpg6w\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.937859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.937830    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istiod-ca-cert\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.938525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.938502    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istiod-ca-cert\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.938746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.938726    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-data\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.939270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.939248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-certs\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.943910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.943830    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-envoy\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.948453 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.948426    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-credential-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.967958 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.966740    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-podinfo\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.995493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.995456    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-token\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:51.999782 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:51.999758    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpg6w\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-kube-api-access-rpg6w\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:52.081155 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:52.081123    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35d6134_28bb_4351_943d_0bd7d643dbe9.slice/crio-af5aa59569792e4ed416a82f76c567d9dd9dffe5a9068673439b6da4423bc35a WatchSource:0}: Error finding container af5aa59569792e4ed416a82f76c567d9dd9dffe5a9068673439b6da4423bc35a: Status 404 returned error can't find the container with id af5aa59569792e4ed416a82f76c567d9dd9dffe5a9068673439b6da4423bc35a
Nov 02 21:45:52.083418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.083395    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"]
Nov 02 21:45:52.083965 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.083936    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:45:52.084030 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.083995    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:45:52.084030 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.084025    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:45:52.280415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.280387    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:52.700472 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:45:52.700427    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebad026_b9bc_4ca1_8626_79ec06333aeb.slice/crio-e1a97722071578f72ee4c40254745aa5138aac7cfb931f0f93ab99424d4bb60c WatchSource:0}: Error finding container e1a97722071578f72ee4c40254745aa5138aac7cfb931f0f93ab99424d4bb60c: Status 404 returned error can't find the container with id e1a97722071578f72ee4c40254745aa5138aac7cfb931f0f93ab99424d4bb60c
Nov 02 21:45:52.703204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.703170    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:45:52.703324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.703206    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"]
Nov 02 21:45:52.703324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.703237    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:45:52.703324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.703265    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:45:52.836456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.836396    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" event={"ID":"0ebad026-b9bc-4ca1-8626-79ec06333aeb","Type":"ContainerStarted","Data":"154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82"}
Nov 02 21:45:52.836633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.836485    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" event={"ID":"0ebad026-b9bc-4ca1-8626-79ec06333aeb","Type":"ContainerStarted","Data":"e1a97722071578f72ee4c40254745aa5138aac7cfb931f0f93ab99424d4bb60c"}
Nov 02 21:45:52.838560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.838525    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" event={"ID":"f35d6134-28bb-4351-943d-0bd7d643dbe9","Type":"ContainerStarted","Data":"bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb"}
Nov 02 21:45:52.838686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.838579    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" event={"ID":"f35d6134-28bb-4351-943d-0bd7d643dbe9","Type":"ContainerStarted","Data":"af5aa59569792e4ed416a82f76c567d9dd9dffe5a9068673439b6da4423bc35a"}
Nov 02 21:45:52.980525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:52.980436    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" podStartSLOduration=1.980421787 podStartE2EDuration="1.980421787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:52.97783 +0000 UTC m=+2240.241366331" watchObservedRunningTime="2026-11-02 21:45:52.980421787 +0000 UTC m=+2240.243958103"
Nov 02 21:45:53.280520 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:53.280488    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:53.286115 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:53.286086    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:53.404888 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:53.404840    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" podStartSLOduration=2.404827038 podStartE2EDuration="2.404827038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:53.098979464 +0000 UTC m=+2240.362515782" watchObservedRunningTime="2026-11-02 21:45:53.404827038 +0000 UTC m=+2240.668363354"
Nov 02 21:45:53.641006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:53.640930    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:53.646443 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:53.646416    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:53.845188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:53.845154    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:53.845544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:53.845356    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:45:53.846508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:53.846486    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:45:53.846636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:45:53.846549    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:46:00.670504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.670079    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"]
Nov 02 21:46:00.676484 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.676456    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:00.733934 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.733907    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"]
Nov 02 21:46:00.777023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.776990    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45\" (UID: \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:00.777179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.777162    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzlz5\" (UniqueName: \"kubernetes.io/projected/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-kube-api-access-hzlz5\") pod \"payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45\" (UID: \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:00.781118 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.781088    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-aa8aff-dockercfg-pppjx\""
Nov 02 21:46:00.781207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.781121    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-aa8aff\""
Nov 02 21:46:00.878682 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.878650    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzlz5\" (UniqueName: \"kubernetes.io/projected/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-kube-api-access-hzlz5\") pod \"payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45\" (UID: \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:00.878862 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.878748    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45\" (UID: \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:00.879457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.879440    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45\" (UID: \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:00.936599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.936533    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzlz5\" (UniqueName: \"kubernetes.io/projected/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-kube-api-access-hzlz5\") pod \"payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45\" (UID: \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:00.992760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:00.992737    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:01.364438 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:46:01.364398    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5834a1eb_f3fc_45a9_a974_4717a6b46b8f.slice/crio-434caffde51ea971feb4257f0d0b0d64606620f844e921ca7f1fc3de800ec3c8 WatchSource:0}: Error finding container 434caffde51ea971feb4257f0d0b0d64606620f844e921ca7f1fc3de800ec3c8: Status 404 returned error can't find the container with id 434caffde51ea971feb4257f0d0b0d64606620f844e921ca7f1fc3de800ec3c8
Nov 02 21:46:01.367410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:01.367386    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"]
Nov 02 21:46:01.896026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:01.895902    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45" event={"ID":"5834a1eb-f3fc-45a9-a974-4717a6b46b8f","Type":"ContainerStarted","Data":"354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944"}
Nov 02 21:46:01.896026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:01.895968    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45" event={"ID":"5834a1eb-f3fc-45a9-a974-4717a6b46b8f","Type":"ContainerStarted","Data":"434caffde51ea971feb4257f0d0b0d64606620f844e921ca7f1fc3de800ec3c8"}
Nov 02 21:46:01.896026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:01.895995    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:02.010592 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:02.010546    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45" podStartSLOduration=2.010533507 podStartE2EDuration="2.010533507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:02.009586958 +0000 UTC m=+2249.273123276" watchObservedRunningTime="2026-11-02 21:46:02.010533507 +0000 UTC m=+2249.274069823"
Nov 02 21:46:04.937571 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:04.937537    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"]
Nov 02 21:46:04.941659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:04.941639    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:04.996760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:04.996710    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"]
Nov 02 21:46:05.047419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.047389    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cc0d6a\""
Nov 02 21:46:05.047419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.047395    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cc0d6a-dockercfg-lfzgc\""
Nov 02 21:46:05.057051 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.057023    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6n4\" (UniqueName: \"kubernetes.io/projected/53adbd40-c61f-444e-a2ea-905cb140f56b-kube-api-access-xw6n4\") pod \"payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5\" (UID: \"53adbd40-c61f-444e-a2ea-905cb140f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:05.057212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.057092    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53adbd40-c61f-444e-a2ea-905cb140f56b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5\" (UID: \"53adbd40-c61f-444e-a2ea-905cb140f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:05.158802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.158734    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6n4\" (UniqueName: \"kubernetes.io/projected/53adbd40-c61f-444e-a2ea-905cb140f56b-kube-api-access-xw6n4\") pod \"payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5\" (UID: \"53adbd40-c61f-444e-a2ea-905cb140f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:05.159049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.158886    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53adbd40-c61f-444e-a2ea-905cb140f56b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5\" (UID: \"53adbd40-c61f-444e-a2ea-905cb140f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:05.160136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.159640    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53adbd40-c61f-444e-a2ea-905cb140f56b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5\" (UID: \"53adbd40-c61f-444e-a2ea-905cb140f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:05.217158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.217068    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6n4\" (UniqueName: \"kubernetes.io/projected/53adbd40-c61f-444e-a2ea-905cb140f56b-kube-api-access-xw6n4\") pod \"payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5\" (UID: \"53adbd40-c61f-444e-a2ea-905cb140f56b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:05.255051 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.255018    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:05.622541 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:46:05.622513    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53adbd40_c61f_444e_a2ea_905cb140f56b.slice/crio-ce00c5aa517df84311681d80c01721d9fc9f85a51428dd242d4b561f6be4239f WatchSource:0}: Error finding container ce00c5aa517df84311681d80c01721d9fc9f85a51428dd242d4b561f6be4239f: Status 404 returned error can't find the container with id ce00c5aa517df84311681d80c01721d9fc9f85a51428dd242d4b561f6be4239f
Nov 02 21:46:05.625571 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.625541    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"]
Nov 02 21:46:05.922676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.922586    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5" event={"ID":"53adbd40-c61f-444e-a2ea-905cb140f56b","Type":"ContainerStarted","Data":"357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0"}
Nov 02 21:46:05.922676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.922635    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5" event={"ID":"53adbd40-c61f-444e-a2ea-905cb140f56b","Type":"ContainerStarted","Data":"ce00c5aa517df84311681d80c01721d9fc9f85a51428dd242d4b561f6be4239f"}
Nov 02 21:46:05.922676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:05.922662    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:06.035799 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:06.035746    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5" podStartSLOduration=2.035733782 podStartE2EDuration="2.035733782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:06.035394706 +0000 UTC m=+2253.298931024" watchObservedRunningTime="2026-11-02 21:46:06.035733782 +0000 UTC m=+2253.299270167"
Nov 02 21:46:12.914288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:12.914260    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:46:14.113050 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:46:14.113018    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:46:14.756210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:14.756172    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:46:16.930903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:16.930875    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:46:24.247238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:24.247177    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k_779a4c9f-0b02-4c72-93e9-c66c28892202/payload-processing/0.log"
Nov 02 21:46:24.528596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:24.528571    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pwkmv_53e705a3-6036-4fdd-800c-90b4eb36f5a8/payload-processing/0.log"
Nov 02 21:46:24.758099 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:24.758072    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pwkmv_53e705a3-6036-4fdd-800c-90b4eb36f5a8/payload-processing/0.log"
Nov 02 21:46:24.969176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:24.969090    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz_fd241a8f-8995-4933-b606-f057639dde3d/payload-processing/0.log"
Nov 02 21:46:25.215569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:25.215538    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz_fd241a8f-8995-4933-b606-f057639dde3d/payload-processing/0.log"
Nov 02 21:46:34.720616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:34.720586    4528 scope.go:122] "RemoveContainer" containerID="55817c94e6890c8c5d5f3f17ae0d14ffdbc5c0a6e7da57e0a94b8394bfec3fcf"
Nov 02 21:46:44.115369 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:46:44.115330    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:46:53.192134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:53.192042    4528 generic.go:356] "Generic (PLEG): container finished" podID="6bf178a9-eb59-4865-be00-c22072fe8439" containerID="ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf" exitCode=0
Nov 02 21:46:53.192511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:53.192138    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" event={"ID":"6bf178a9-eb59-4865-be00-c22072fe8439","Type":"ContainerDied","Data":"ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf"}
Nov 02 21:46:53.192562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:53.192521    4528 scope.go:122] "RemoveContainer" containerID="ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf"
Nov 02 21:46:53.193960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:53.193930    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:46:53.194067 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:53.194013    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:46:53.194067 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:53.194057    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:46:54.199939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:54.199902    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" event={"ID":"6bf178a9-eb59-4865-be00-c22072fe8439","Type":"ContainerStarted","Data":"6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88"}
Nov 02 21:46:54.421555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:54.421515    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:46:54.421965 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:54.421939    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:46:54.422163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:54.422136    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 21:46:55.420134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:55.418418    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:46:55.420134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:55.418475    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 21:46:56.422521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:56.422482    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:46:56.423103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:56.423069    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 21:46:57.423242 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:57.423203    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:46:57.423833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:57.423803    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 21:46:58.418473 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:58.418435    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 21:46:58.418768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:58.418735    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 21:46:59.417603 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:59.417576    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:46:59.418084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:59.417966    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:46:59.418554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:46:59.418537    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:47:07.605732 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.605701    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"]
Nov 02 21:47:07.609802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.609782    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.662564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.662534    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:47:07.662683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.662599    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:47:07.664566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.664544    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"]
Nov 02 21:47:07.673292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.673266    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40723a0c-s6szl\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.673399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.673299    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2c94\" (UniqueName: \"kubernetes.io/projected/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-kube-api-access-h2c94\") pod \"maas-api-revoke-sub-40723a0c-s6szl\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.673445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.673395    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40723a0c-s6szl\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.688948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.688918    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:47:07.689053 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.688935    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:47:07.775566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.774867    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40723a0c-s6szl\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.775566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.774922    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2c94\" (UniqueName: \"kubernetes.io/projected/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-kube-api-access-h2c94\") pod \"maas-api-revoke-sub-40723a0c-s6szl\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.775566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.774998    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40723a0c-s6szl\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.779400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.779372    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40723a0c-s6szl\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.783604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.783548    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40723a0c-s6szl\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.836215 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.836188    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2c94\" (UniqueName: \"kubernetes.io/projected/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-kube-api-access-h2c94\") pod \"maas-api-revoke-sub-40723a0c-s6szl\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:07.882471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.882403    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"]
Nov 02 21:47:07.922424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:07.922398    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:08.305608 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:47:08.305568    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c11505_b89a_4d53_aee8_b5ee3ad990f2.slice/crio-86b173f5558ea74e46ef453d4c681a8d7cbeced89bf90086989eaf9a328e98c3 WatchSource:0}: Error finding container 86b173f5558ea74e46ef453d4c681a8d7cbeced89bf90086989eaf9a328e98c3: Status 404 returned error can't find the container with id 86b173f5558ea74e46ef453d4c681a8d7cbeced89bf90086989eaf9a328e98c3
Nov 02 21:47:08.310379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:08.310338    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"]
Nov 02 21:47:08.337570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:08.337536    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl" event={"ID":"a5c11505-b89a-4d53-aee8-b5ee3ad990f2","Type":"ContainerStarted","Data":"86b173f5558ea74e46ef453d4c681a8d7cbeced89bf90086989eaf9a328e98c3"}
Nov 02 21:47:09.345072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:09.345035    4528 generic.go:356] "Generic (PLEG): container finished" podID="a5c11505-b89a-4d53-aee8-b5ee3ad990f2" containerID="32a26e267e1b554d222a27db328345dc801047320ad6bca06a13541fc2e47387" exitCode=0
Nov 02 21:47:09.345491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:09.345140    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl" event={"ID":"a5c11505-b89a-4d53-aee8-b5ee3ad990f2","Type":"ContainerDied","Data":"32a26e267e1b554d222a27db328345dc801047320ad6bca06a13541fc2e47387"}
Nov 02 21:47:10.538696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.538667    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:10.628059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.627979    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-kube-api-access-h2c94\" (UniqueName: \"kubernetes.io/projected/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-kube-api-access-h2c94\") pod \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") "
Nov 02 21:47:10.628224 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.628061    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-xks-ca\") pod \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") "
Nov 02 21:47:10.628224 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.628151    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-openshift-ca\") pod \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\" (UID: \"a5c11505-b89a-4d53-aee8-b5ee3ad990f2\") "
Nov 02 21:47:10.628522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.628500    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-openshift-ca" pod "a5c11505-b89a-4d53-aee8-b5ee3ad990f2" (UID: "a5c11505-b89a-4d53-aee8-b5ee3ad990f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:47:10.628699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.628682    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:10.630027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.630002    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-kube-api-access-h2c94" pod "a5c11505-b89a-4d53-aee8-b5ee3ad990f2" (UID: "a5c11505-b89a-4d53-aee8-b5ee3ad990f2"). InnerVolumeSpecName "kube-api-access-h2c94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:47:10.630150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.630039    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-xks-ca" pod "a5c11505-b89a-4d53-aee8-b5ee3ad990f2" (UID: "a5c11505-b89a-4d53-aee8-b5ee3ad990f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:47:10.729568 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.729537    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2c94\" (UniqueName: \"kubernetes.io/projected/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-kube-api-access-h2c94\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:10.729568 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:10.729564    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c11505-b89a-4d53-aee8-b5ee3ad990f2-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:11.147728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:11.147696    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:47:11.357814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:11.357789    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"
Nov 02 21:47:11.357983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:11.357828    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl" event={"ID":"a5c11505-b89a-4d53-aee8-b5ee3ad990f2","Type":"ContainerDied","Data":"86b173f5558ea74e46ef453d4c681a8d7cbeced89bf90086989eaf9a328e98c3"}
Nov 02 21:47:11.357983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:11.357875    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86b173f5558ea74e46ef453d4c681a8d7cbeced89bf90086989eaf9a328e98c3"
Nov 02 21:47:12.883340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:12.883293    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc" podUID="698428c7-1a90-460d-8afc-b96ad903c217" containerName="payload-processing" containerID="cri-o://758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715" gracePeriod=25
Nov 02 21:47:13.135713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.135649    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:47:13.272823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.272782    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698428c7-1a90-460d-8afc-b96ad903c217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698428c7-1a90-460d-8afc-b96ad903c217-plugins-config-volume\") pod \"698428c7-1a90-460d-8afc-b96ad903c217\" (UID: \"698428c7-1a90-460d-8afc-b96ad903c217\") "
Nov 02 21:47:13.273041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.273022    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698428c7-1a90-460d-8afc-b96ad903c217-kube-api-access-ggk5x\" (UniqueName: \"kubernetes.io/projected/698428c7-1a90-460d-8afc-b96ad903c217-kube-api-access-ggk5x\") pod \"698428c7-1a90-460d-8afc-b96ad903c217\" (UID: \"698428c7-1a90-460d-8afc-b96ad903c217\") "
Nov 02 21:47:13.273175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.273155    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698428c7-1a90-460d-8afc-b96ad903c217-plugins-config-volume" pod "698428c7-1a90-460d-8afc-b96ad903c217" (UID: "698428c7-1a90-460d-8afc-b96ad903c217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:47:13.273536 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.273514    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/698428c7-1a90-460d-8afc-b96ad903c217-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:13.275073 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.275053    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698428c7-1a90-460d-8afc-b96ad903c217-kube-api-access-ggk5x" pod "698428c7-1a90-460d-8afc-b96ad903c217" (UID: "698428c7-1a90-460d-8afc-b96ad903c217"). InnerVolumeSpecName "kube-api-access-ggk5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:47:13.374481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.374448    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggk5x\" (UniqueName: \"kubernetes.io/projected/698428c7-1a90-460d-8afc-b96ad903c217-kube-api-access-ggk5x\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:13.375206 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.375178    4528 generic.go:356] "Generic (PLEG): container finished" podID="698428c7-1a90-460d-8afc-b96ad903c217" containerID="758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715" exitCode=0
Nov 02 21:47:13.375343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.375230    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"
Nov 02 21:47:13.375343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.375234    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc" event={"ID":"698428c7-1a90-460d-8afc-b96ad903c217","Type":"ContainerDied","Data":"758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715"}
Nov 02 21:47:13.375343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.375341    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc" event={"ID":"698428c7-1a90-460d-8afc-b96ad903c217","Type":"ContainerDied","Data":"5e23152421654a7d71a7bf19f00e6be2e1e667b6f170855e73d85422967f1471"}
Nov 02 21:47:13.375518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.375366    4528 scope.go:122] "RemoveContainer" containerID="758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715"
Nov 02 21:47:13.386778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.386754    4528 scope.go:122] "RemoveContainer" containerID="758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715"
Nov 02 21:47:13.387084 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:47:13.387064    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715\": container with ID starting with 758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715 not found: ID does not exist" containerID="758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715"
Nov 02 21:47:13.387213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.387089    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715"} err="failed to get container status \"758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715\": rpc error: code = NotFound desc = could not find container \"758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715\": container with ID starting with 758f2d519a89cfb9ec68658f5f86bdbb968012a6eea1b789d28a9bbba9751715 not found: ID does not exist"
Nov 02 21:47:13.539749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.539722    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"]
Nov 02 21:47:13.542331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:13.542311    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-7dd5c6bb5-blsbc"]
Nov 02 21:47:14.113307 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:47:14.113270    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:47:14.913102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:14.913072    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698428c7-1a90-460d-8afc-b96ad903c217" path="/var/lib/kubelet/pods/698428c7-1a90-460d-8afc-b96ad903c217/volumes"
Nov 02 21:47:39.665720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:39.647921    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": read tcp 10.133.0.2:53810->10.133.0.196:15021: read: connection reset by peer" start-of-body=
Nov 02 21:47:39.665720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:39.647989    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": read tcp 10.133.0.2:53810->10.133.0.196:15021: read: connection reset by peer"
Nov 02 21:47:39.843334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:39.843300    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": read tcp 10.133.0.2:36778->10.133.0.197:15021: read: connection reset by peer" start-of-body=
Nov 02 21:47:39.843463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:39.843382    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": read tcp 10.133.0.2:36778->10.133.0.197:15021: read: connection reset by peer"
Nov 02 21:47:40.614307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.614275    4528 generic.go:356] "Generic (PLEG): container finished" podID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerID="154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82" exitCode=0
Nov 02 21:47:40.614504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.614353    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" event={"ID":"0ebad026-b9bc-4ca1-8626-79ec06333aeb","Type":"ContainerDied","Data":"154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82"}
Nov 02 21:47:40.614808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.614780    4528 scope.go:122] "RemoveContainer" containerID="154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82"
Nov 02 21:47:40.616350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.616324    4528 generic.go:356] "Generic (PLEG): container finished" podID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerID="bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb" exitCode=0
Nov 02 21:47:40.616449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.616365    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:47:40.616449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.616396    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" event={"ID":"f35d6134-28bb-4351-943d-0bd7d643dbe9","Type":"ContainerDied","Data":"bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb"}
Nov 02 21:47:40.616449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.616436    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:47:40.616549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.616474    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:47:40.616730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.616716    4528 scope.go:122] "RemoveContainer" containerID="bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb"
Nov 02 21:47:40.618245 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.618219    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:47:40.618326 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.618311    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:47:40.618375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:40.618361    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:47:41.650155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:41.650030    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" event={"ID":"f35d6134-28bb-4351-943d-0bd7d643dbe9","Type":"ContainerStarted","Data":"c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e"}
Nov 02 21:47:41.656698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:41.656659    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" event={"ID":"0ebad026-b9bc-4ca1-8626-79ec06333aeb","Type":"ContainerStarted","Data":"77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18"}
Nov 02 21:47:42.280725 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:42.280693    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:47:42.280921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:42.280832    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 02 21:47:42.280921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:42.280885    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 02 21:47:42.650205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:42.646453    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:47:42.650205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:42.646567    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 21:47:42.650205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:42.646625    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 21:47:43.280645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:43.280609    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 02 21:47:43.280828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:43.280668    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 02 21:47:43.652192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:43.644499    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 21:47:43.652192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:43.644571    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 21:47:44.157651 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:47:44.157566    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:47:44.280851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:44.280812    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 02 21:47:44.281005 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:44.280872    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 02 21:47:44.644134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:44.643579    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 21:47:44.644134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:44.643640    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 21:47:45.281706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:45.281673    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 02 21:47:45.282144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:45.281741    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 02 21:47:45.644189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:45.643505    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 21:47:45.644189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:45.643566    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 21:47:46.040129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:46.039318    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:47:46.281045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:46.281003    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 02 21:47:46.281267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:46.281121    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 02 21:47:46.649131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:46.648465    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 21:47:46.649131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:46.648529    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 21:47:47.284641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:47.284614    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:47:47.285150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:47.285135    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:47:47.285813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:47.285792    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:47:47.645390 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:47.645317    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:47:47.645742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:47.645720    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:47:47.646418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:47.646401    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:47:50.508727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.508692    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"]
Nov 02 21:47:50.509230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.508976    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k" podUID="779a4c9f-0b02-4c72-93e9-c66c28892202" containerName="payload-processing" containerID="cri-o://6d51bbb31191d0ffd172dae9c8655593dec8214f74649f9a01cd1cfadb505a5a" gracePeriod=30
Nov 02 21:47:50.565666 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.565642    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"]
Nov 02 21:47:50.565849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.565829    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4" podUID="38362c6b-9f43-41e2-9174-deb14ea99a4f" containerName="payload-pre-processing" containerID="cri-o://5747033cc13927f766a865485764afea532d78a4a235d45b6b1fb19c2ce0b15b" gracePeriod=30
Nov 02 21:47:50.712645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.712542    4528 generic.go:356] "Generic (PLEG): container finished" podID="38362c6b-9f43-41e2-9174-deb14ea99a4f" containerID="5747033cc13927f766a865485764afea532d78a4a235d45b6b1fb19c2ce0b15b" exitCode=0
Nov 02 21:47:50.712645 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.712608    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4" event={"ID":"38362c6b-9f43-41e2-9174-deb14ea99a4f","Type":"ContainerDied","Data":"5747033cc13927f766a865485764afea532d78a4a235d45b6b1fb19c2ce0b15b"}
Nov 02 21:47:50.714531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.714506    4528 generic.go:356] "Generic (PLEG): container finished" podID="779a4c9f-0b02-4c72-93e9-c66c28892202" containerID="6d51bbb31191d0ffd172dae9c8655593dec8214f74649f9a01cd1cfadb505a5a" exitCode=0
Nov 02 21:47:50.714846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.714822    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k" event={"ID":"779a4c9f-0b02-4c72-93e9-c66c28892202","Type":"ContainerDied","Data":"6d51bbb31191d0ffd172dae9c8655593dec8214f74649f9a01cd1cfadb505a5a"}
Nov 02 21:47:50.844405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.844383    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:47:50.868437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.868415    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:47:50.915564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.915535    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/779a4c9f-0b02-4c72-93e9-c66c28892202-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779a4c9f-0b02-4c72-93e9-c66c28892202-plugins-config-volume\") pod \"779a4c9f-0b02-4c72-93e9-c66c28892202\" (UID: \"779a4c9f-0b02-4c72-93e9-c66c28892202\") "
Nov 02 21:47:50.915749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.915729    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779a4c9f-0b02-4c72-93e9-c66c28892202-kube-api-access-fqfkp\" (UniqueName: \"kubernetes.io/projected/779a4c9f-0b02-4c72-93e9-c66c28892202-kube-api-access-fqfkp\") pod \"779a4c9f-0b02-4c72-93e9-c66c28892202\" (UID: \"779a4c9f-0b02-4c72-93e9-c66c28892202\") "
Nov 02 21:47:50.915888 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.915871    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779a4c9f-0b02-4c72-93e9-c66c28892202-plugins-config-volume" pod "779a4c9f-0b02-4c72-93e9-c66c28892202" (UID: "779a4c9f-0b02-4c72-93e9-c66c28892202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:47:50.916074 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.916060    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/779a4c9f-0b02-4c72-93e9-c66c28892202-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:50.917741 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:50.917720    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779a4c9f-0b02-4c72-93e9-c66c28892202-kube-api-access-fqfkp" pod "779a4c9f-0b02-4c72-93e9-c66c28892202" (UID: "779a4c9f-0b02-4c72-93e9-c66c28892202"). InnerVolumeSpecName "kube-api-access-fqfkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:47:51.016943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.016899    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38362c6b-9f43-41e2-9174-deb14ea99a4f-kube-api-access-kwsms\" (UniqueName: \"kubernetes.io/projected/38362c6b-9f43-41e2-9174-deb14ea99a4f-kube-api-access-kwsms\") pod \"38362c6b-9f43-41e2-9174-deb14ea99a4f\" (UID: \"38362c6b-9f43-41e2-9174-deb14ea99a4f\") "
Nov 02 21:47:51.017162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.016988    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38362c6b-9f43-41e2-9174-deb14ea99a4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38362c6b-9f43-41e2-9174-deb14ea99a4f-plugins-config-volume\") pod \"38362c6b-9f43-41e2-9174-deb14ea99a4f\" (UID: \"38362c6b-9f43-41e2-9174-deb14ea99a4f\") "
Nov 02 21:47:51.017507 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.017486    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfkp\" (UniqueName: \"kubernetes.io/projected/779a4c9f-0b02-4c72-93e9-c66c28892202-kube-api-access-fqfkp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:51.017922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.017855    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38362c6b-9f43-41e2-9174-deb14ea99a4f-plugins-config-volume" pod "38362c6b-9f43-41e2-9174-deb14ea99a4f" (UID: "38362c6b-9f43-41e2-9174-deb14ea99a4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:47:51.020166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.020088    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38362c6b-9f43-41e2-9174-deb14ea99a4f-kube-api-access-kwsms" pod "38362c6b-9f43-41e2-9174-deb14ea99a4f" (UID: "38362c6b-9f43-41e2-9174-deb14ea99a4f"). InnerVolumeSpecName "kube-api-access-kwsms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:47:51.153130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.125440    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwsms\" (UniqueName: \"kubernetes.io/projected/38362c6b-9f43-41e2-9174-deb14ea99a4f-kube-api-access-kwsms\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:51.162922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.162895    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38362c6b-9f43-41e2-9174-deb14ea99a4f-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:51.721256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.721229    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"
Nov 02 21:47:51.721688 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.721271    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k" event={"ID":"779a4c9f-0b02-4c72-93e9-c66c28892202","Type":"ContainerDied","Data":"a5ea0b866c12e4538ef2add9062c27b4d28f5c423403c65cf5872caba76dcfc8"}
Nov 02 21:47:51.721688 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.721333    4528 scope.go:122] "RemoveContainer" containerID="6d51bbb31191d0ffd172dae9c8655593dec8214f74649f9a01cd1cfadb505a5a"
Nov 02 21:47:51.722848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.722825    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4" event={"ID":"38362c6b-9f43-41e2-9174-deb14ea99a4f","Type":"ContainerDied","Data":"71af9afd376eee6c4c722ac0c3a56930bc52cc4c719b27e22fd7d7ce411b2c4f"}
Nov 02 21:47:51.722923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.722862    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"
Nov 02 21:47:51.730768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.730747    4528 scope.go:122] "RemoveContainer" containerID="5747033cc13927f766a865485764afea532d78a4a235d45b6b1fb19c2ce0b15b"
Nov 02 21:47:51.886421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.886341    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"]
Nov 02 21:47:51.890261 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.890236    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e12f99-54c88bcbb6-tqt5k"]
Nov 02 21:47:51.937276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.937244    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"]
Nov 02 21:47:51.937447 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:51.937427    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk" podUID="ef571e2a-0892-492e-a3d8-d57dd8fe5358" containerName="payload-processing" containerID="cri-o://92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e" gracePeriod=30
Nov 02 21:47:52.000948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.000921    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"]
Nov 02 21:47:52.003485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.003465    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e12f99-7cf88f6959-xb6g4"]
Nov 02 21:47:52.019996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.019971    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"]
Nov 02 21:47:52.020209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.020188    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm" podUID="26caed4f-a41c-4e8d-8170-36ae88f6b177" containerName="payload-pre-processing" containerID="cri-o://d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154" gracePeriod=30
Nov 02 21:47:52.289333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.289304    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:47:52.302427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.302405    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:47:52.382298 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.382261    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef571e2a-0892-492e-a3d8-d57dd8fe5358-kube-api-access-77ss2\" (UniqueName: \"kubernetes.io/projected/ef571e2a-0892-492e-a3d8-d57dd8fe5358-kube-api-access-77ss2\") pod \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\" (UID: \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\") "
Nov 02 21:47:52.382440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.382333    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26caed4f-a41c-4e8d-8170-36ae88f6b177-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26caed4f-a41c-4e8d-8170-36ae88f6b177-plugins-config-volume\") pod \"26caed4f-a41c-4e8d-8170-36ae88f6b177\" (UID: \"26caed4f-a41c-4e8d-8170-36ae88f6b177\") "
Nov 02 21:47:52.382440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.382379    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef571e2a-0892-492e-a3d8-d57dd8fe5358-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef571e2a-0892-492e-a3d8-d57dd8fe5358-plugins-config-volume\") pod \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\" (UID: \"ef571e2a-0892-492e-a3d8-d57dd8fe5358\") "
Nov 02 21:47:52.382440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.382422    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26caed4f-a41c-4e8d-8170-36ae88f6b177-kube-api-access-d7vzq\" (UniqueName: \"kubernetes.io/projected/26caed4f-a41c-4e8d-8170-36ae88f6b177-kube-api-access-d7vzq\") pod \"26caed4f-a41c-4e8d-8170-36ae88f6b177\" (UID: \"26caed4f-a41c-4e8d-8170-36ae88f6b177\") "
Nov 02 21:47:52.382726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.382701    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26caed4f-a41c-4e8d-8170-36ae88f6b177-plugins-config-volume" pod "26caed4f-a41c-4e8d-8170-36ae88f6b177" (UID: "26caed4f-a41c-4e8d-8170-36ae88f6b177"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:47:52.382815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.382722    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef571e2a-0892-492e-a3d8-d57dd8fe5358-plugins-config-volume" pod "ef571e2a-0892-492e-a3d8-d57dd8fe5358" (UID: "ef571e2a-0892-492e-a3d8-d57dd8fe5358"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:47:52.384488 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.384468    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26caed4f-a41c-4e8d-8170-36ae88f6b177-kube-api-access-d7vzq" pod "26caed4f-a41c-4e8d-8170-36ae88f6b177" (UID: "26caed4f-a41c-4e8d-8170-36ae88f6b177"). InnerVolumeSpecName "kube-api-access-d7vzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:47:52.384566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.384520    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef571e2a-0892-492e-a3d8-d57dd8fe5358-kube-api-access-77ss2" pod "ef571e2a-0892-492e-a3d8-d57dd8fe5358" (UID: "ef571e2a-0892-492e-a3d8-d57dd8fe5358"). InnerVolumeSpecName "kube-api-access-77ss2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:47:52.484163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.484053    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26caed4f-a41c-4e8d-8170-36ae88f6b177-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:52.484163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.484081    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef571e2a-0892-492e-a3d8-d57dd8fe5358-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:52.484163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.484095    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7vzq\" (UniqueName: \"kubernetes.io/projected/26caed4f-a41c-4e8d-8170-36ae88f6b177-kube-api-access-d7vzq\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:52.484163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.484136    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77ss2\" (UniqueName: \"kubernetes.io/projected/ef571e2a-0892-492e-a3d8-d57dd8fe5358-kube-api-access-77ss2\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:47:52.729755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.729720    4528 generic.go:356] "Generic (PLEG): container finished" podID="ef571e2a-0892-492e-a3d8-d57dd8fe5358" containerID="92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e" exitCode=0
Nov 02 21:47:52.730238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.729765    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"
Nov 02 21:47:52.730238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.729809    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk" event={"ID":"ef571e2a-0892-492e-a3d8-d57dd8fe5358","Type":"ContainerDied","Data":"92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e"}
Nov 02 21:47:52.730238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.729865    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk" event={"ID":"ef571e2a-0892-492e-a3d8-d57dd8fe5358","Type":"ContainerDied","Data":"608db8965f30a6728a0b95afcbe6fbbc61aa62bb02b0f56858be669d1d8043c1"}
Nov 02 21:47:52.730238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.729899    4528 scope.go:122] "RemoveContainer" containerID="92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e"
Nov 02 21:47:52.734024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.734003    4528 generic.go:356] "Generic (PLEG): container finished" podID="26caed4f-a41c-4e8d-8170-36ae88f6b177" containerID="d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154" exitCode=0
Nov 02 21:47:52.734148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.734042    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm" event={"ID":"26caed4f-a41c-4e8d-8170-36ae88f6b177","Type":"ContainerDied","Data":"d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154"}
Nov 02 21:47:52.734148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.734055    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"
Nov 02 21:47:52.734148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.734080    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm" event={"ID":"26caed4f-a41c-4e8d-8170-36ae88f6b177","Type":"ContainerDied","Data":"49b23964e6909a53fc2ab56ddf739e5807c0efed46bd603cbfef3f773cdf8af7"}
Nov 02 21:47:52.741654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.741555    4528 scope.go:122] "RemoveContainer" containerID="92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e"
Nov 02 21:47:52.742623 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:47:52.741909    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e\": container with ID starting with 92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e not found: ID does not exist" containerID="92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e"
Nov 02 21:47:52.742623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.741942    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e"} err="failed to get container status \"92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e\": rpc error: code = NotFound desc = could not find container \"92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e\": container with ID starting with 92f670e8f39ab3b9a77de31c82c221dba532556980cdd00f1acc152d5d3af96e not found: ID does not exist"
Nov 02 21:47:52.742623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.741963    4528 scope.go:122] "RemoveContainer" containerID="d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154"
Nov 02 21:47:52.752433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.752417    4528 scope.go:122] "RemoveContainer" containerID="d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154"
Nov 02 21:47:52.752660 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:47:52.752646    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154\": container with ID starting with d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154 not found: ID does not exist" containerID="d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154"
Nov 02 21:47:52.752706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.752667    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154"} err="failed to get container status \"d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154\": rpc error: code = NotFound desc = could not find container \"d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154\": container with ID starting with d312a7dc74d76bec9cf844018aad8eb082f95a411068b275b3e3918f9699a154 not found: ID does not exist"
Nov 02 21:47:52.894476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.894449    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"]
Nov 02 21:47:52.897327 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.897303    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ce2453-84d7dffbcf-qfkbm"]
Nov 02 21:47:52.912923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.912894    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26caed4f-a41c-4e8d-8170-36ae88f6b177" path="/var/lib/kubelet/pods/26caed4f-a41c-4e8d-8170-36ae88f6b177/volumes"
Nov 02 21:47:52.913306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.913293    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38362c6b-9f43-41e2-9174-deb14ea99a4f" path="/var/lib/kubelet/pods/38362c6b-9f43-41e2-9174-deb14ea99a4f/volumes"
Nov 02 21:47:52.913625 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:52.913614    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779a4c9f-0b02-4c72-93e9-c66c28892202" path="/var/lib/kubelet/pods/779a4c9f-0b02-4c72-93e9-c66c28892202/volumes"
Nov 02 21:47:53.006721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:53.006638    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"]
Nov 02 21:47:53.010550 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:53.010524    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-7df5584f9d-6vgrk"]
Nov 02 21:47:54.914102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:54.914059    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef571e2a-0892-492e-a3d8-d57dd8fe5358" path="/var/lib/kubelet/pods/ef571e2a-0892-492e-a3d8-d57dd8fe5358/volumes"
Nov 02 21:47:58.184568 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:58.184537    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"]
Nov 02 21:47:58.185023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:47:58.184740    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy" containerID="cri-o://1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a" gracePeriod=30
Nov 02 21:48:01.580626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:01.580590    4528 patch_prober.go:29] interesting pod/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:48:01.581007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:01.580647    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:48:03.145569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.145532    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"]
Nov 02 21:48:03.146419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146395    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ef571e2a-0892-492e-a3d8-d57dd8fe5358" containerName="payload-processing"
Nov 02 21:48:03.146419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146420    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef571e2a-0892-492e-a3d8-d57dd8fe5358" containerName="payload-processing"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146443    4528 cpu_manager.go:414] "containerMap: removing container" podUID="779a4c9f-0b02-4c72-93e9-c66c28892202" containerName="payload-processing"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146448    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779a4c9f-0b02-4c72-93e9-c66c28892202" containerName="payload-processing"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146457    4528 cpu_manager.go:414] "containerMap: removing container" podUID="26caed4f-a41c-4e8d-8170-36ae88f6b177" containerName="payload-pre-processing"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146463    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26caed4f-a41c-4e8d-8170-36ae88f6b177" containerName="payload-pre-processing"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146497    4528 cpu_manager.go:414] "containerMap: removing container" podUID="38362c6b-9f43-41e2-9174-deb14ea99a4f" containerName="payload-pre-processing"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146503    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38362c6b-9f43-41e2-9174-deb14ea99a4f" containerName="payload-pre-processing"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146518    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a5c11505-b89a-4d53-aee8-b5ee3ad990f2" containerName="revoke-keys"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146523    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c11505-b89a-4d53-aee8-b5ee3ad990f2" containerName="revoke-keys"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146536    4528 cpu_manager.go:414] "containerMap: removing container" podUID="698428c7-1a90-460d-8afc-b96ad903c217" containerName="payload-processing"
Nov 02 21:48:03.146598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146541    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698428c7-1a90-460d-8afc-b96ad903c217" containerName="payload-processing"
Nov 02 21:48:03.147104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146632    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="779a4c9f-0b02-4c72-93e9-c66c28892202" containerName="payload-processing"
Nov 02 21:48:03.147104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146647    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="26caed4f-a41c-4e8d-8170-36ae88f6b177" containerName="payload-pre-processing"
Nov 02 21:48:03.147104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146654    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="38362c6b-9f43-41e2-9174-deb14ea99a4f" containerName="payload-pre-processing"
Nov 02 21:48:03.147104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146663    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c11505-b89a-4d53-aee8-b5ee3ad990f2" containerName="revoke-keys"
Nov 02 21:48:03.147104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146670    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="698428c7-1a90-460d-8afc-b96ad903c217" containerName="payload-processing"
Nov 02 21:48:03.147104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.146676    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef571e2a-0892-492e-a3d8-d57dd8fe5358" containerName="payload-processing"
Nov 02 21:48:03.150624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.150593    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:03.198313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.198101    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"]
Nov 02 21:48:03.206629 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.206604    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmmz\" (UniqueName: \"kubernetes.io/projected/58e0764a-0f38-4904-ad72-f6c2c51c6840-kube-api-access-jdmmz\") pod \"payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79\" (UID: \"58e0764a-0f38-4904-ad72-f6c2c51c6840\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:03.206748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.206660    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e0764a-0f38-4904-ad72-f6c2c51c6840-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79\" (UID: \"58e0764a-0f38-4904-ad72-f6c2c51c6840\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:03.243263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.243233    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-942164-dockercfg-bsv6w\""
Nov 02 21:48:03.243263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.243234    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-942164\""
Nov 02 21:48:03.307807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.307772    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e0764a-0f38-4904-ad72-f6c2c51c6840-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79\" (UID: \"58e0764a-0f38-4904-ad72-f6c2c51c6840\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:03.308062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.308040    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmmz\" (UniqueName: \"kubernetes.io/projected/58e0764a-0f38-4904-ad72-f6c2c51c6840-kube-api-access-jdmmz\") pod \"payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79\" (UID: \"58e0764a-0f38-4904-ad72-f6c2c51c6840\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:03.308407 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.308388    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e0764a-0f38-4904-ad72-f6c2c51c6840-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79\" (UID: \"58e0764a-0f38-4904-ad72-f6c2c51c6840\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:03.365445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.365412    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmmz\" (UniqueName: \"kubernetes.io/projected/58e0764a-0f38-4904-ad72-f6c2c51c6840-kube-api-access-jdmmz\") pod \"payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79\" (UID: \"58e0764a-0f38-4904-ad72-f6c2c51c6840\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:03.452344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.452322    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:48:03.469777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.469759    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:03.510760 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.510729    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-token\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-token\") pod \"3734a586-cc32-4840-af2a-0d7d4b63bc46\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") "
Nov 02 21:48:03.510914 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.510834    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-podinfo\") pod \"3734a586-cc32-4840-af2a-0d7d4b63bc46\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") "
Nov 02 21:48:03.510914 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.510872    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-credential-socket\") pod \"3734a586-cc32-4840-af2a-0d7d4b63bc46\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") "
Nov 02 21:48:03.511049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.510981    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3734a586-cc32-4840-af2a-0d7d4b63bc46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3734a586-cc32-4840-af2a-0d7d4b63bc46-istiod-ca-cert\") pod \"3734a586-cc32-4840-af2a-0d7d4b63bc46\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") "
Nov 02 21:48:03.511049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511029    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-data\") pod \"3734a586-cc32-4840-af2a-0d7d4b63bc46\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") "
Nov 02 21:48:03.511192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511066    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-kube-api-access-4sfqp\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-kube-api-access-4sfqp\") pod \"3734a586-cc32-4840-af2a-0d7d4b63bc46\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") "
Nov 02 21:48:03.511192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511142    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-socket\") pod \"3734a586-cc32-4840-af2a-0d7d4b63bc46\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") "
Nov 02 21:48:03.511192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511173    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-envoy\") pod \"3734a586-cc32-4840-af2a-0d7d4b63bc46\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") "
Nov 02 21:48:03.511338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511201    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-certs\") pod \"3734a586-cc32-4840-af2a-0d7d4b63bc46\" (UID: \"3734a586-cc32-4840-af2a-0d7d4b63bc46\") "
Nov 02 21:48:03.511338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511275    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-data" pod "3734a586-cc32-4840-af2a-0d7d4b63bc46" (UID: "3734a586-cc32-4840-af2a-0d7d4b63bc46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:48:03.511433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511352    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3734a586-cc32-4840-af2a-0d7d4b63bc46-istiod-ca-cert" pod "3734a586-cc32-4840-af2a-0d7d4b63bc46" (UID: "3734a586-cc32-4840-af2a-0d7d4b63bc46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:48:03.511643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511610    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-certs" pod "3734a586-cc32-4840-af2a-0d7d4b63bc46" (UID: "3734a586-cc32-4840-af2a-0d7d4b63bc46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:48:03.511779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511646    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3734a586-cc32-4840-af2a-0d7d4b63bc46-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:03.511779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511667    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:03.511779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511680    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-socket" pod "3734a586-cc32-4840-af2a-0d7d4b63bc46" (UID: "3734a586-cc32-4840-af2a-0d7d4b63bc46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:48:03.511779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.511736    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-credential-socket" pod "3734a586-cc32-4840-af2a-0d7d4b63bc46" (UID: "3734a586-cc32-4840-af2a-0d7d4b63bc46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:48:03.512987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.512964    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-token" pod "3734a586-cc32-4840-af2a-0d7d4b63bc46" (UID: "3734a586-cc32-4840-af2a-0d7d4b63bc46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:48:03.513266 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.513246    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-envoy" pod "3734a586-cc32-4840-af2a-0d7d4b63bc46" (UID: "3734a586-cc32-4840-af2a-0d7d4b63bc46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:48:03.513638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.513621    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-kube-api-access-4sfqp" pod "3734a586-cc32-4840-af2a-0d7d4b63bc46" (UID: "3734a586-cc32-4840-af2a-0d7d4b63bc46"). InnerVolumeSpecName "kube-api-access-4sfqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:48:03.513698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.513624    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-podinfo" pod "3734a586-cc32-4840-af2a-0d7d4b63bc46" (UID: "3734a586-cc32-4840-af2a-0d7d4b63bc46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:48:03.613837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.613805    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfqp\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-kube-api-access-4sfqp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:03.613837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.613836    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:03.614050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.613849    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:03.614050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.613861    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:03.614050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.613872    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:03.614050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.613883    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3734a586-cc32-4840-af2a-0d7d4b63bc46-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:03.614050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.613895    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3734a586-cc32-4840-af2a-0d7d4b63bc46-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:03.809188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.809152    4528 generic.go:356] "Generic (PLEG): container finished" podID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerID="1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a" exitCode=0
Nov 02 21:48:03.809383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.809207    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"
Nov 02 21:48:03.809383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.809229    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" event={"ID":"3734a586-cc32-4840-af2a-0d7d4b63bc46","Type":"ContainerDied","Data":"1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a"}
Nov 02 21:48:03.809383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.809282    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl" event={"ID":"3734a586-cc32-4840-af2a-0d7d4b63bc46","Type":"ContainerDied","Data":"0da1068b3ddc5fa7b43bf6b7216be777d6ba8ac2a9eb9e0588504fb741ea1cc0"}
Nov 02 21:48:03.809383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.809306    4528 scope.go:122] "RemoveContainer" containerID="1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a"
Nov 02 21:48:03.818681 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.818659    4528 scope.go:122] "RemoveContainer" containerID="4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c"
Nov 02 21:48:03.827132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.827099    4528 scope.go:122] "RemoveContainer" containerID="1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a"
Nov 02 21:48:03.827421 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:48:03.827401    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a\": container with ID starting with 1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a not found: ID does not exist" containerID="1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a"
Nov 02 21:48:03.827493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.827427    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a"} err="failed to get container status \"1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a\": rpc error: code = NotFound desc = could not find container \"1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a\": container with ID starting with 1245ab93e9ac56c7bb30343798129a94cb1e84014e7f3fafd362bdf15b14897a not found: ID does not exist"
Nov 02 21:48:03.827493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.827446    4528 scope.go:122] "RemoveContainer" containerID="4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c"
Nov 02 21:48:03.827673 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:48:03.827656    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c\": container with ID starting with 4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c not found: ID does not exist" containerID="4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c"
Nov 02 21:48:03.827714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.827678    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c"} err="failed to get container status \"4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c\": rpc error: code = NotFound desc = could not find container \"4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c\": container with ID starting with 4b163c87f9806ebefc7094c56440b757f7cb84c209b59ef12387f0abdea7c07c not found: ID does not exist"
Nov 02 21:48:03.834644 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:48:03.834623    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e0764a_0f38_4904_ad72_f6c2c51c6840.slice/crio-3bcfa2231332f7a5dc6edad065deed7f8986587ca6405b892bcae0ffca13203f WatchSource:0}: Error finding container 3bcfa2231332f7a5dc6edad065deed7f8986587ca6405b892bcae0ffca13203f: Status 404 returned error can't find the container with id 3bcfa2231332f7a5dc6edad065deed7f8986587ca6405b892bcae0ffca13203f
Nov 02 21:48:03.838271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.838249    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"]
Nov 02 21:48:03.975180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.975141    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"]
Nov 02 21:48:03.978105 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:03.978086    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e12f99-openshift-default-77c8d78fbc-c5gkl"]
Nov 02 21:48:04.816770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:04.816726    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79" event={"ID":"58e0764a-0f38-4904-ad72-f6c2c51c6840","Type":"ContainerStarted","Data":"59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973"}
Nov 02 21:48:04.817237 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:04.816795    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:04.817237 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:04.816840    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79" event={"ID":"58e0764a-0f38-4904-ad72-f6c2c51c6840","Type":"ContainerStarted","Data":"3bcfa2231332f7a5dc6edad065deed7f8986587ca6405b892bcae0ffca13203f"}
Nov 02 21:48:04.913565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:04.913533    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" path="/var/lib/kubelet/pods/3734a586-cc32-4840-af2a-0d7d4b63bc46/volumes"
Nov 02 21:48:04.931987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:04.931934    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79" podStartSLOduration=1.931918016 podStartE2EDuration="1.931918016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:04.931048365 +0000 UTC m=+2372.194584682" watchObservedRunningTime="2026-11-02 21:48:04.931918016 +0000 UTC m=+2372.195454336"
Nov 02 21:48:12.400636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.400606    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"]
Nov 02 21:48:12.401244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.401229    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy"
Nov 02 21:48:12.401298 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.401246    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy"
Nov 02 21:48:12.401338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.401332    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy"
Nov 02 21:48:12.401375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.401343    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy"
Nov 02 21:48:12.405882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.405867    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:12.457634 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.457604    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"]
Nov 02 21:48:12.505050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.505021    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa60bc\""
Nov 02 21:48:12.505214 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.505075    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aa60bc-dockercfg-5pkmc\""
Nov 02 21:48:12.529179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.529148    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfd5s\" (UniqueName: \"kubernetes.io/projected/5e9cd98f-4abe-43e0-9d90-113f90279a37-kube-api-access-bfd5s\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d\" (UID: \"5e9cd98f-4abe-43e0-9d90-113f90279a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:12.529367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.529350    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9cd98f-4abe-43e0-9d90-113f90279a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d\" (UID: \"5e9cd98f-4abe-43e0-9d90-113f90279a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:12.630238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.630209    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfd5s\" (UniqueName: \"kubernetes.io/projected/5e9cd98f-4abe-43e0-9d90-113f90279a37-kube-api-access-bfd5s\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d\" (UID: \"5e9cd98f-4abe-43e0-9d90-113f90279a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:12.630399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.630324    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9cd98f-4abe-43e0-9d90-113f90279a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d\" (UID: \"5e9cd98f-4abe-43e0-9d90-113f90279a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:12.630950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.630931    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9cd98f-4abe-43e0-9d90-113f90279a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d\" (UID: \"5e9cd98f-4abe-43e0-9d90-113f90279a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:12.687313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.687249    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfd5s\" (UniqueName: \"kubernetes.io/projected/5e9cd98f-4abe-43e0-9d90-113f90279a37-kube-api-access-bfd5s\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d\" (UID: \"5e9cd98f-4abe-43e0-9d90-113f90279a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:12.718684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:12.718663    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:13.084760 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:48:13.084731    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9cd98f_4abe_43e0_9d90_113f90279a37.slice/crio-2e3de284625bdab53cee5c8fb3fdb4a6c73d633b44132c8aac4b58ae538c58bd WatchSource:0}: Error finding container 2e3de284625bdab53cee5c8fb3fdb4a6c73d633b44132c8aac4b58ae538c58bd: Status 404 returned error can't find the container with id 2e3de284625bdab53cee5c8fb3fdb4a6c73d633b44132c8aac4b58ae538c58bd
Nov 02 21:48:13.087093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.087069    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"]
Nov 02 21:48:13.186146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.184859    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"]
Nov 02 21:48:13.186146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.185462    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy"
Nov 02 21:48:13.186146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.185475    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3734a586-cc32-4840-af2a-0d7d4b63bc46" containerName="istio-proxy"
Nov 02 21:48:13.188953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.188934    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:13.241880 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.241853    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"]
Nov 02 21:48:13.337678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.337598    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r\" (UID: \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:13.337813 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.337753    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptcz\" (UniqueName: \"kubernetes.io/projected/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-kube-api-access-2ptcz\") pod \"payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r\" (UID: \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:13.438876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.438844    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptcz\" (UniqueName: \"kubernetes.io/projected/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-kube-api-access-2ptcz\") pod \"payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r\" (UID: \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:13.439281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.438951    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r\" (UID: \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:13.439650 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.439632    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r\" (UID: \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:13.496009 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.495971    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptcz\" (UniqueName: \"kubernetes.io/projected/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-kube-api-access-2ptcz\") pod \"payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r\" (UID: \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:13.502814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.502793    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:13.871936 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:48:13.871907    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3abfa7c_0ad4_4832_a1ee_550d78c25c6a.slice/crio-d281f14487e4c551f9b9a2f2455e0bc54e171bebb55067a28cdaf379d8698cd1 WatchSource:0}: Error finding container d281f14487e4c551f9b9a2f2455e0bc54e171bebb55067a28cdaf379d8698cd1: Status 404 returned error can't find the container with id d281f14487e4c551f9b9a2f2455e0bc54e171bebb55067a28cdaf379d8698cd1
Nov 02 21:48:13.873576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.873549    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"]
Nov 02 21:48:13.896697 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.896667    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d" event={"ID":"5e9cd98f-4abe-43e0-9d90-113f90279a37","Type":"ContainerStarted","Data":"6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a"}
Nov 02 21:48:13.896834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.896728    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d" event={"ID":"5e9cd98f-4abe-43e0-9d90-113f90279a37","Type":"ContainerStarted","Data":"2e3de284625bdab53cee5c8fb3fdb4a6c73d633b44132c8aac4b58ae538c58bd"}
Nov 02 21:48:13.897163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.897141    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:13.898059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:13.898036    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r" event={"ID":"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a","Type":"ContainerStarted","Data":"d281f14487e4c551f9b9a2f2455e0bc54e171bebb55067a28cdaf379d8698cd1"}
Nov 02 21:48:14.013366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:14.013311    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d" podStartSLOduration=2.013295376 podStartE2EDuration="2.013295376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:14.009856186 +0000 UTC m=+2381.273392502" watchObservedRunningTime="2026-11-02 21:48:14.013295376 +0000 UTC m=+2381.276831692"
Nov 02 21:48:14.115751 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:48:14.115717    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:48:14.905488 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:14.905405    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r" event={"ID":"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a","Type":"ContainerStarted","Data":"261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686"}
Nov 02 21:48:14.905488 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:14.905473    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:15.021333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:15.021285    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r" podStartSLOduration=2.021271679 podStartE2EDuration="2.021271679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:15.019537576 +0000 UTC m=+2382.283073894" watchObservedRunningTime="2026-11-02 21:48:15.021271679 +0000 UTC m=+2382.284807996"
Nov 02 21:48:15.826780 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:15.826753    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:48:24.915530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:24.915488    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:48:25.922016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:25.921980    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:48:34.868294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:34.868262    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:48:38.406686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:38.406651    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"]
Nov 02 21:48:38.514215 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:38.514181    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"]
Nov 02 21:48:43.406964 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.406928    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29" podUID="1e98f242-a7ba-4ddb-bb03-9f740e1670f1" containerName="payload-processing" containerID="cri-o://8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea" gracePeriod=25
Nov 02 21:48:43.514980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.514927    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw" podUID="bb43cdb9-4948-4b75-87a3-03151929ef51" containerName="payload-pre-processing" containerID="cri-o://65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1" gracePeriod=25
Nov 02 21:48:43.655755 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:48:43.655734    4528 logging.go:55] [core] [Channel #5143 SubChannel #5144]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.173:50052", ServerName: "10.133.0.173:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.173:50052: connect: connection refused"
Nov 02 21:48:43.679118 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.679044    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:48:43.784421 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.784394    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:48:43.813082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.813053    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-kube-api-access-gdcck\" (UniqueName: \"kubernetes.io/projected/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-kube-api-access-gdcck\") pod \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\" (UID: \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\") "
Nov 02 21:48:43.813246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.813228    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-plugins-config-volume\") pod \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\" (UID: \"1e98f242-a7ba-4ddb-bb03-9f740e1670f1\") "
Nov 02 21:48:43.813555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.813532    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-plugins-config-volume" pod "1e98f242-a7ba-4ddb-bb03-9f740e1670f1" (UID: "1e98f242-a7ba-4ddb-bb03-9f740e1670f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:48:43.814994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.814973    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-kube-api-access-gdcck" pod "1e98f242-a7ba-4ddb-bb03-9f740e1670f1" (UID: "1e98f242-a7ba-4ddb-bb03-9f740e1670f1"). InnerVolumeSpecName "kube-api-access-gdcck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:48:43.913927 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.913894    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb43cdb9-4948-4b75-87a3-03151929ef51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43cdb9-4948-4b75-87a3-03151929ef51-plugins-config-volume\") pod \"bb43cdb9-4948-4b75-87a3-03151929ef51\" (UID: \"bb43cdb9-4948-4b75-87a3-03151929ef51\") "
Nov 02 21:48:43.914144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.913984    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb43cdb9-4948-4b75-87a3-03151929ef51-kube-api-access-wfmtn\" (UniqueName: \"kubernetes.io/projected/bb43cdb9-4948-4b75-87a3-03151929ef51-kube-api-access-wfmtn\") pod \"bb43cdb9-4948-4b75-87a3-03151929ef51\" (UID: \"bb43cdb9-4948-4b75-87a3-03151929ef51\") "
Nov 02 21:48:43.914389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.914330    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb43cdb9-4948-4b75-87a3-03151929ef51-plugins-config-volume" pod "bb43cdb9-4948-4b75-87a3-03151929ef51" (UID: "bb43cdb9-4948-4b75-87a3-03151929ef51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:48:43.914616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.914592    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdcck\" (UniqueName: \"kubernetes.io/projected/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-kube-api-access-gdcck\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:43.914746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.914620    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e98f242-a7ba-4ddb-bb03-9f740e1670f1-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:43.914746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.914636    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43cdb9-4948-4b75-87a3-03151929ef51-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:43.916533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:43.916510    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb43cdb9-4948-4b75-87a3-03151929ef51-kube-api-access-wfmtn" pod "bb43cdb9-4948-4b75-87a3-03151929ef51" (UID: "bb43cdb9-4948-4b75-87a3-03151929ef51"). InnerVolumeSpecName "kube-api-access-wfmtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:48:44.016103 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.016075    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfmtn\" (UniqueName: \"kubernetes.io/projected/bb43cdb9-4948-4b75-87a3-03151929ef51-kube-api-access-wfmtn\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:48:44.099158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.099093    4528 generic.go:356] "Generic (PLEG): container finished" podID="bb43cdb9-4948-4b75-87a3-03151929ef51" containerID="65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1" exitCode=0
Nov 02 21:48:44.099307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.099169    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw" event={"ID":"bb43cdb9-4948-4b75-87a3-03151929ef51","Type":"ContainerDied","Data":"65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1"}
Nov 02 21:48:44.099307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.099233    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw" event={"ID":"bb43cdb9-4948-4b75-87a3-03151929ef51","Type":"ContainerDied","Data":"a5b775ad225f0d2655af6a92276dc27182bd12f6b43ba7a5163639cebcc57762"}
Nov 02 21:48:44.099307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.099256    4528 scope.go:122] "RemoveContainer" containerID="65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1"
Nov 02 21:48:44.099307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.099184    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"
Nov 02 21:48:44.100828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.100799    4528 generic.go:356] "Generic (PLEG): container finished" podID="1e98f242-a7ba-4ddb-bb03-9f740e1670f1" containerID="8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea" exitCode=0
Nov 02 21:48:44.100951 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.100847    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"
Nov 02 21:48:44.100951 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.100878    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29" event={"ID":"1e98f242-a7ba-4ddb-bb03-9f740e1670f1","Type":"ContainerDied","Data":"8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea"}
Nov 02 21:48:44.101088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.100961    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29" event={"ID":"1e98f242-a7ba-4ddb-bb03-9f740e1670f1","Type":"ContainerDied","Data":"3148f4577dcbc74158b80059502014a7c17ee3a36f1b58d2d8e04910d07cea21"}
Nov 02 21:48:44.110693 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.110672    4528 scope.go:122] "RemoveContainer" containerID="65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1"
Nov 02 21:48:44.111028 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:48:44.111007    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1\": container with ID starting with 65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1 not found: ID does not exist" containerID="65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1"
Nov 02 21:48:44.111084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.111036    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1"} err="failed to get container status \"65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1\": rpc error: code = NotFound desc = could not find container \"65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1\": container with ID starting with 65448b76d67dcf761c62e31b8ab3b28cb6582fb49032b8ae230f8c322f7ef2f1 not found: ID does not exist"
Nov 02 21:48:44.111084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.111054    4528 scope.go:122] "RemoveContainer" containerID="8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea"
Nov 02 21:48:44.115504 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:48:44.115482    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:48:44.120471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.120455    4528 scope.go:122] "RemoveContainer" containerID="8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea"
Nov 02 21:48:44.120714 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:48:44.120697    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea\": container with ID starting with 8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea not found: ID does not exist" containerID="8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea"
Nov 02 21:48:44.120778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.120725    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea"} err="failed to get container status \"8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea\": rpc error: code = NotFound desc = could not find container \"8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea\": container with ID starting with 8ec7a9822cbb3a311cca221abcfb448be09c56d70b025cb8ccef66c24e3261ea not found: ID does not exist"
Nov 02 21:48:44.263489 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.263455    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"]
Nov 02 21:48:44.268564 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.268504    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw"]
Nov 02 21:48:44.376871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.376837    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"]
Nov 02 21:48:44.380180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.380154    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29"]
Nov 02 21:48:44.655994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.655905    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-be6df0-548d66df4f-rcz29" podUID="1e98f242-a7ba-4ddb-bb03-9f740e1670f1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.174:50052\" within 1s: context deadline exceeded"
Nov 02 21:48:44.655994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.655925    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be6df0-555d7d5695-glfvw" podUID="bb43cdb9-4948-4b75-87a3-03151929ef51" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.173:50052\" within 1s: context deadline exceeded"
Nov 02 21:48:44.655994 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:48:44.655960    4528 logging.go:55] [core] [Channel #5142 SubChannel #5145]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.174:50052", ServerName: "10.133.0.174:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.174:50052: operation was canceled"
Nov 02 21:48:44.913508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.913436    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e98f242-a7ba-4ddb-bb03-9f740e1670f1" path="/var/lib/kubelet/pods/1e98f242-a7ba-4ddb-bb03-9f740e1670f1/volumes"
Nov 02 21:48:44.913807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:48:44.913794    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb43cdb9-4948-4b75-87a3-03151929ef51" path="/var/lib/kubelet/pods/bb43cdb9-4948-4b75-87a3-03151929ef51/volumes"
Nov 02 21:49:09.083445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.083413    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"]
Nov 02 21:49:09.083965 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.083641    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz" podUID="fd241a8f-8995-4933-b606-f057639dde3d" containerName="payload-processing" containerID="cri-o://d346c1c473bff8a8f945ce433c9d6eec15c34103c3da6cafbdc4c40fe0d0423d" gracePeriod=30
Nov 02 21:49:09.138057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.138023    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"]
Nov 02 21:49:09.138309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.138263    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj" podUID="6d835758-48be-4b27-b62b-7769662df352" containerName="payload-pre-processing" containerID="cri-o://8b99132b2cd54c4147e758a0d99f0e5b23009e4d3e00dfb87293c6eddeb1551f" gracePeriod=30
Nov 02 21:49:09.256700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.256646    4528 generic.go:356] "Generic (PLEG): container finished" podID="fd241a8f-8995-4933-b606-f057639dde3d" containerID="d346c1c473bff8a8f945ce433c9d6eec15c34103c3da6cafbdc4c40fe0d0423d" exitCode=0
Nov 02 21:49:09.256779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.256716    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz" event={"ID":"fd241a8f-8995-4933-b606-f057639dde3d","Type":"ContainerDied","Data":"d346c1c473bff8a8f945ce433c9d6eec15c34103c3da6cafbdc4c40fe0d0423d"}
Nov 02 21:49:09.259601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.259576    4528 generic.go:356] "Generic (PLEG): container finished" podID="6d835758-48be-4b27-b62b-7769662df352" containerID="8b99132b2cd54c4147e758a0d99f0e5b23009e4d3e00dfb87293c6eddeb1551f" exitCode=0
Nov 02 21:49:09.259732 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.259649    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj" event={"ID":"6d835758-48be-4b27-b62b-7769662df352","Type":"ContainerDied","Data":"8b99132b2cd54c4147e758a0d99f0e5b23009e4d3e00dfb87293c6eddeb1551f"}
Nov 02 21:49:09.406514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.406483    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:49:09.442830 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.442808    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:49:09.495975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.495945    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd241a8f-8995-4933-b606-f057639dde3d-kube-api-access-7chcs\" (UniqueName: \"kubernetes.io/projected/fd241a8f-8995-4933-b606-f057639dde3d-kube-api-access-7chcs\") pod \"fd241a8f-8995-4933-b606-f057639dde3d\" (UID: \"fd241a8f-8995-4933-b606-f057639dde3d\") "
Nov 02 21:49:09.496137 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.496061    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd241a8f-8995-4933-b606-f057639dde3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd241a8f-8995-4933-b606-f057639dde3d-plugins-config-volume\") pod \"fd241a8f-8995-4933-b606-f057639dde3d\" (UID: \"fd241a8f-8995-4933-b606-f057639dde3d\") "
Nov 02 21:49:09.496418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.496393    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd241a8f-8995-4933-b606-f057639dde3d-plugins-config-volume" pod "fd241a8f-8995-4933-b606-f057639dde3d" (UID: "fd241a8f-8995-4933-b606-f057639dde3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:49:09.497932 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.497908    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd241a8f-8995-4933-b606-f057639dde3d-kube-api-access-7chcs" pod "fd241a8f-8995-4933-b606-f057639dde3d" (UID: "fd241a8f-8995-4933-b606-f057639dde3d"). InnerVolumeSpecName "kube-api-access-7chcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:49:09.597347 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.597312    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d835758-48be-4b27-b62b-7769662df352-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d835758-48be-4b27-b62b-7769662df352-plugins-config-volume\") pod \"6d835758-48be-4b27-b62b-7769662df352\" (UID: \"6d835758-48be-4b27-b62b-7769662df352\") "
Nov 02 21:49:09.597525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.597365    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d835758-48be-4b27-b62b-7769662df352-kube-api-access-6r6rs\" (UniqueName: \"kubernetes.io/projected/6d835758-48be-4b27-b62b-7769662df352-kube-api-access-6r6rs\") pod \"6d835758-48be-4b27-b62b-7769662df352\" (UID: \"6d835758-48be-4b27-b62b-7769662df352\") "
Nov 02 21:49:09.597664 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.597641    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d835758-48be-4b27-b62b-7769662df352-plugins-config-volume" pod "6d835758-48be-4b27-b62b-7769662df352" (UID: "6d835758-48be-4b27-b62b-7769662df352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:49:09.597863 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.597839    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d835758-48be-4b27-b62b-7769662df352-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:09.597863 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.597862    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7chcs\" (UniqueName: \"kubernetes.io/projected/fd241a8f-8995-4933-b606-f057639dde3d-kube-api-access-7chcs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:09.598022 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.597875    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd241a8f-8995-4933-b606-f057639dde3d-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:09.599331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.599312    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d835758-48be-4b27-b62b-7769662df352-kube-api-access-6r6rs" pod "6d835758-48be-4b27-b62b-7769662df352" (UID: "6d835758-48be-4b27-b62b-7769662df352"). InnerVolumeSpecName "kube-api-access-6r6rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:49:09.698861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:09.698765    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r6rs\" (UniqueName: \"kubernetes.io/projected/6d835758-48be-4b27-b62b-7769662df352-kube-api-access-6r6rs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:10.267556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.267530    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"
Nov 02 21:49:10.267985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.267526    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz" event={"ID":"fd241a8f-8995-4933-b606-f057639dde3d","Type":"ContainerDied","Data":"ec5e11a67fe5c42fd8f600ae2f942d3d368d3b64284e3cedde77ae38abb7a00e"}
Nov 02 21:49:10.267985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.267654    4528 scope.go:122] "RemoveContainer" containerID="d346c1c473bff8a8f945ce433c9d6eec15c34103c3da6cafbdc4c40fe0d0423d"
Nov 02 21:49:10.269773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.269745    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj" event={"ID":"6d835758-48be-4b27-b62b-7769662df352","Type":"ContainerDied","Data":"28bb7b310403040d2d038480e2a07d17fa0d21f08ebee256a900736970d25562"}
Nov 02 21:49:10.269923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.269754    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"
Nov 02 21:49:10.279377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.279359    4528 scope.go:122] "RemoveContainer" containerID="8b99132b2cd54c4147e758a0d99f0e5b23009e4d3e00dfb87293c6eddeb1551f"
Nov 02 21:49:10.436017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.435915    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"]
Nov 02 21:49:10.484525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.484483    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-6dc87b4687-xzzvj"]
Nov 02 21:49:10.486965 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.486941    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"]
Nov 02 21:49:10.487193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.487172    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r" podUID="a3abfa7c-0ad4-4832-a1ee-550d78c25c6a" containerName="payload-processing" containerID="cri-o://261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686" gracePeriod=30
Nov 02 21:49:10.538486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.538461    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"]
Nov 02 21:49:10.538646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.538629    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d" podUID="5e9cd98f-4abe-43e0-9d90-113f90279a37" containerName="payload-pre-processing" containerID="cri-o://6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a" gracePeriod=30
Nov 02 21:49:10.550541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.550517    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"]
Nov 02 21:49:10.556343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.556323    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-6bb574cccb-k4tsz"]
Nov 02 21:49:10.809100 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.809077    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:49:10.818401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.818374    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:49:10.914809 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.914775    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d835758-48be-4b27-b62b-7769662df352" path="/var/lib/kubelet/pods/6d835758-48be-4b27-b62b-7769662df352/volumes"
Nov 02 21:49:10.915250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.915235    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd241a8f-8995-4933-b606-f057639dde3d" path="/var/lib/kubelet/pods/fd241a8f-8995-4933-b606-f057639dde3d/volumes"
Nov 02 21:49:10.919814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.919787    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-plugins-config-volume\") pod \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\" (UID: \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\") "
Nov 02 21:49:10.919960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.919830    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-kube-api-access-2ptcz\" (UniqueName: \"kubernetes.io/projected/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-kube-api-access-2ptcz\") pod \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\" (UID: \"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a\") "
Nov 02 21:49:10.920028 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.919998    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9cd98f-4abe-43e0-9d90-113f90279a37-kube-api-access-bfd5s\" (UniqueName: \"kubernetes.io/projected/5e9cd98f-4abe-43e0-9d90-113f90279a37-kube-api-access-bfd5s\") pod \"5e9cd98f-4abe-43e0-9d90-113f90279a37\" (UID: \"5e9cd98f-4abe-43e0-9d90-113f90279a37\") "
Nov 02 21:49:10.920088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.920038    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9cd98f-4abe-43e0-9d90-113f90279a37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9cd98f-4abe-43e0-9d90-113f90279a37-plugins-config-volume\") pod \"5e9cd98f-4abe-43e0-9d90-113f90279a37\" (UID: \"5e9cd98f-4abe-43e0-9d90-113f90279a37\") "
Nov 02 21:49:10.920231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.920209    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-plugins-config-volume" pod "a3abfa7c-0ad4-4832-a1ee-550d78c25c6a" (UID: "a3abfa7c-0ad4-4832-a1ee-550d78c25c6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:49:10.920589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.920544    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9cd98f-4abe-43e0-9d90-113f90279a37-plugins-config-volume" pod "5e9cd98f-4abe-43e0-9d90-113f90279a37" (UID: "5e9cd98f-4abe-43e0-9d90-113f90279a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:49:10.920589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.920576    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:10.922637 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.922608    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9cd98f-4abe-43e0-9d90-113f90279a37-kube-api-access-bfd5s" pod "5e9cd98f-4abe-43e0-9d90-113f90279a37" (UID: "5e9cd98f-4abe-43e0-9d90-113f90279a37"). InnerVolumeSpecName "kube-api-access-bfd5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:49:10.926241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:10.926220    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-kube-api-access-2ptcz" pod "a3abfa7c-0ad4-4832-a1ee-550d78c25c6a" (UID: "a3abfa7c-0ad4-4832-a1ee-550d78c25c6a"). InnerVolumeSpecName "kube-api-access-2ptcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:49:11.024134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.021962    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfd5s\" (UniqueName: \"kubernetes.io/projected/5e9cd98f-4abe-43e0-9d90-113f90279a37-kube-api-access-bfd5s\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:11.024134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.022005    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9cd98f-4abe-43e0-9d90-113f90279a37-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:11.024134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.022023    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ptcz\" (UniqueName: \"kubernetes.io/projected/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a-kube-api-access-2ptcz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:11.277640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.277556    4528 generic.go:356] "Generic (PLEG): container finished" podID="a3abfa7c-0ad4-4832-a1ee-550d78c25c6a" containerID="261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686" exitCode=0
Nov 02 21:49:11.277640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.277611    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"
Nov 02 21:49:11.278183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.277610    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r" event={"ID":"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a","Type":"ContainerDied","Data":"261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686"}
Nov 02 21:49:11.278183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.277763    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r" event={"ID":"a3abfa7c-0ad4-4832-a1ee-550d78c25c6a","Type":"ContainerDied","Data":"d281f14487e4c551f9b9a2f2455e0bc54e171bebb55067a28cdaf379d8698cd1"}
Nov 02 21:49:11.278183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.277807    4528 scope.go:122] "RemoveContainer" containerID="261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686"
Nov 02 21:49:11.280504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.280477    4528 generic.go:356] "Generic (PLEG): container finished" podID="5e9cd98f-4abe-43e0-9d90-113f90279a37" containerID="6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a" exitCode=0
Nov 02 21:49:11.280638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.280516    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"
Nov 02 21:49:11.280638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.280566    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d" event={"ID":"5e9cd98f-4abe-43e0-9d90-113f90279a37","Type":"ContainerDied","Data":"6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a"}
Nov 02 21:49:11.280638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.280617    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d" event={"ID":"5e9cd98f-4abe-43e0-9d90-113f90279a37","Type":"ContainerDied","Data":"2e3de284625bdab53cee5c8fb3fdb4a6c73d633b44132c8aac4b58ae538c58bd"}
Nov 02 21:49:11.288205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.288186    4528 scope.go:122] "RemoveContainer" containerID="261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686"
Nov 02 21:49:11.288483 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:49:11.288465    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686\": container with ID starting with 261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686 not found: ID does not exist" containerID="261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686"
Nov 02 21:49:11.288542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.288491    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686"} err="failed to get container status \"261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686\": rpc error: code = NotFound desc = could not find container \"261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686\": container with ID starting with 261897bf4f7b75cc62c9864081a7362c30d07f5f1bf1f2eecee99902d4e11686 not found: ID does not exist"
Nov 02 21:49:11.288542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.288508    4528 scope.go:122] "RemoveContainer" containerID="6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a"
Nov 02 21:49:11.296440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.296426    4528 scope.go:122] "RemoveContainer" containerID="6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a"
Nov 02 21:49:11.296675 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:49:11.296659    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a\": container with ID starting with 6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a not found: ID does not exist" containerID="6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a"
Nov 02 21:49:11.296716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.296681    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a"} err="failed to get container status \"6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a\": rpc error: code = NotFound desc = could not find container \"6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a\": container with ID starting with 6c6b7a4cfd5582b7013d6ae21142ded5c94a6ae78be292280b2d5dafdbff721a not found: ID does not exist"
Nov 02 21:49:11.442452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.442421    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"]
Nov 02 21:49:11.445941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.445921    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa60bc-857d9d4fd6-dpp6r"]
Nov 02 21:49:11.554343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.554260    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"]
Nov 02 21:49:11.561312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:11.561290    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-77c9f9f975-j8m9d"]
Nov 02 21:49:12.912871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:12.912842    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9cd98f-4abe-43e0-9d90-113f90279a37" path="/var/lib/kubelet/pods/5e9cd98f-4abe-43e0-9d90-113f90279a37/volumes"
Nov 02 21:49:12.913255 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:12.913247    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3abfa7c-0ad4-4832-a1ee-550d78c25c6a" path="/var/lib/kubelet/pods/a3abfa7c-0ad4-4832-a1ee-550d78c25c6a/volumes"
Nov 02 21:49:14.111995 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:49:14.111959    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:49:15.090161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:15.090128    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:49:18.436220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436190    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"]
Nov 02 21:49:18.436732 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436717    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a3abfa7c-0ad4-4832-a1ee-550d78c25c6a" containerName="payload-processing"
Nov 02 21:49:18.436776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436733    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3abfa7c-0ad4-4832-a1ee-550d78c25c6a" containerName="payload-processing"
Nov 02 21:49:18.436776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436750    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bb43cdb9-4948-4b75-87a3-03151929ef51" containerName="payload-pre-processing"
Nov 02 21:49:18.436776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436756    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb43cdb9-4948-4b75-87a3-03151929ef51" containerName="payload-pre-processing"
Nov 02 21:49:18.436776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436774    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fd241a8f-8995-4933-b606-f057639dde3d" containerName="payload-processing"
Nov 02 21:49:18.436921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436780    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd241a8f-8995-4933-b606-f057639dde3d" containerName="payload-processing"
Nov 02 21:49:18.436921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436788    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5e9cd98f-4abe-43e0-9d90-113f90279a37" containerName="payload-pre-processing"
Nov 02 21:49:18.436921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436792    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9cd98f-4abe-43e0-9d90-113f90279a37" containerName="payload-pre-processing"
Nov 02 21:49:18.436921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436800    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6d835758-48be-4b27-b62b-7769662df352" containerName="payload-pre-processing"
Nov 02 21:49:18.436921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436804    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d835758-48be-4b27-b62b-7769662df352" containerName="payload-pre-processing"
Nov 02 21:49:18.436921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436829    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1e98f242-a7ba-4ddb-bb03-9f740e1670f1" containerName="payload-processing"
Nov 02 21:49:18.436921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436834    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e98f242-a7ba-4ddb-bb03-9f740e1670f1" containerName="payload-processing"
Nov 02 21:49:18.437172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436958    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e98f242-a7ba-4ddb-bb03-9f740e1670f1" containerName="payload-processing"
Nov 02 21:49:18.437172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436971    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3abfa7c-0ad4-4832-a1ee-550d78c25c6a" containerName="payload-processing"
Nov 02 21:49:18.437172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436979    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb43cdb9-4948-4b75-87a3-03151929ef51" containerName="payload-pre-processing"
Nov 02 21:49:18.437172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436987    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd241a8f-8995-4933-b606-f057639dde3d" containerName="payload-processing"
Nov 02 21:49:18.437172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.436995    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9cd98f-4abe-43e0-9d90-113f90279a37" containerName="payload-pre-processing"
Nov 02 21:49:18.437172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.437003    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d835758-48be-4b27-b62b-7769662df352" containerName="payload-pre-processing"
Nov 02 21:49:18.441810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.441793    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.516865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.516811    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-data\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.516865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.516867    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-envoy\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.517250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.516897    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-token\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.517250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.516936    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-podinfo\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.517250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.516961    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbf6236-95c3-4847-9650-f5de5af07ad2-istiod-ca-cert\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.517250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.516987    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-credential-socket\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.517250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.517034    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-socket\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.517250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.517097    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzwtk\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-kube-api-access-jzwtk\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.517250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.517216    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-certs\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.521580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.520637    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-eabda2-openshift-default-dockercfg-lvqtq\""
Nov 02 21:49:18.567194 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.564530    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"]
Nov 02 21:49:18.622039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.621544    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-certs\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.622039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.621708    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-data\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.622039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.621747    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-envoy\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.622039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.621771    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-token\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.622039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.621798    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-podinfo\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.622039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.621821    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbf6236-95c3-4847-9650-f5de5af07ad2-istiod-ca-cert\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.622039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.621848    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-credential-socket\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.622039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.621890    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-socket\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.622039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.621948    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzwtk\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-kube-api-access-jzwtk\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.622667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.622643    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-certs\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.626340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.622871    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-data\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.626765 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.626719    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-envoy\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.628375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.628350    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-credential-socket\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.629231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.629210    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbf6236-95c3-4847-9650-f5de5af07ad2-istiod-ca-cert\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.629633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.629610    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-socket\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.630171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.630146    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-podinfo\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.679142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.679103    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzwtk\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-kube-api-access-jzwtk\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.683586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.683565    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-token\") pod \"e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") " pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:18.758871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:18.758801    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:19.177318 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:49:19.177278    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbf6236_95c3_4847_9650_f5de5af07ad2.slice/crio-21df70c30fbc5ab2f3c63ef473ee556e55d515409a8c4ec2df867a0274ae67ba WatchSource:0}: Error finding container 21df70c30fbc5ab2f3c63ef473ee556e55d515409a8c4ec2df867a0274ae67ba: Status 404 returned error can't find the container with id 21df70c30fbc5ab2f3c63ef473ee556e55d515409a8c4ec2df867a0274ae67ba
Nov 02 21:49:19.180002 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.179970    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:49:19.180104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.180033    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"]
Nov 02 21:49:19.180104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.180035    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:49:19.180223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.180152    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:49:19.339540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.339506    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" event={"ID":"8cbf6236-95c3-4847-9650-f5de5af07ad2","Type":"ContainerStarted","Data":"78323832e5ffd012c315bf361e38803f14d376ab18f74471d764ca2407edef06"}
Nov 02 21:49:19.339707 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.339563    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" event={"ID":"8cbf6236-95c3-4847-9650-f5de5af07ad2","Type":"ContainerStarted","Data":"21df70c30fbc5ab2f3c63ef473ee556e55d515409a8c4ec2df867a0274ae67ba"}
Nov 02 21:49:19.459591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.459497    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" podStartSLOduration=1.45948389 podStartE2EDuration="1.45948389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:19.457791883 +0000 UTC m=+2446.721328238" watchObservedRunningTime="2026-11-02 21:49:19.45948389 +0000 UTC m=+2446.723020207"
Nov 02 21:49:19.518551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.518517    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"]
Nov 02 21:49:19.518779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.518756    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" containerID="cri-o://439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178" gracePeriod=30
Nov 02 21:49:19.759861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:19.759778    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:20.291842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:20.291807    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"]
Nov 02 21:49:20.410433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:20.410402    4528 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:49:20.410609 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:20.410459    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:49:20.764263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:20.764229    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:21.351982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:21.351944    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:21.352920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:21.352894    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:49:24.788223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.788151    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:49:24.933196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933167    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-certs\") pod \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") "
Nov 02 21:49:24.933343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933214    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-socket\") pod \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") "
Nov 02 21:49:24.933343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933274    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istiod-ca-cert\") pod \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") "
Nov 02 21:49:24.933343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933313    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-podinfo\") pod \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") "
Nov 02 21:49:24.933519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933348    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-data\") pod \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") "
Nov 02 21:49:24.933519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933400    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-certs" pod "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" (UID: "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:49:24.933519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933508    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-kube-api-access-v8khc\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-kube-api-access-v8khc\") pod \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") "
Nov 02 21:49:24.933685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933540    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-credential-socket\") pod \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") "
Nov 02 21:49:24.933685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933545    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-socket" pod "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" (UID: "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:49:24.933685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933585    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-envoy\") pod \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") "
Nov 02 21:49:24.933685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933612    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-token\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-token\") pod \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\" (UID: \"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef\") "
Nov 02 21:49:24.933685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933661    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-data" pod "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" (UID: "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:49:24.933945 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933681    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istiod-ca-cert" pod "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" (UID: "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:49:24.933945 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.933835    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-credential-socket" pod "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" (UID: "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:49:24.934089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.934070    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:24.934165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.934095    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:24.934165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.934124    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:24.934165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.934140    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:24.934165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.934153    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:24.935586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.935557    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-envoy" pod "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" (UID: "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:49:24.935700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.935586    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-podinfo" pod "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" (UID: "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:49:24.935700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.935648    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-kube-api-access-v8khc" pod "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" (UID: "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef"). InnerVolumeSpecName "kube-api-access-v8khc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:49:24.935700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:24.935669    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-token" pod "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" (UID: "b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:49:25.035178 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.035144    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8khc\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-kube-api-access-v8khc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:25.035178 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.035178    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:25.035368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.035193    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:25.035368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.035208    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:25.293141 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.293092    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5" podUID="53adbd40-c61f-444e-a2ea-905cb140f56b" containerName="payload-pre-processing" containerID="cri-o://357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0" gracePeriod=25
Nov 02 21:49:25.375283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.375255    4528 generic.go:356] "Generic (PLEG): container finished" podID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerID="439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178" exitCode=0
Nov 02 21:49:25.375411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.375306    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"
Nov 02 21:49:25.375411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.375337    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" event={"ID":"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef","Type":"ContainerDied","Data":"439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178"}
Nov 02 21:49:25.375411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.375388    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q" event={"ID":"b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef","Type":"ContainerDied","Data":"e20f10f4e87847d174b8bcf7dedd9422d0a4a2a7ea905da4eeb37825589956b5"}
Nov 02 21:49:25.375411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.375412    4528 scope.go:122] "RemoveContainer" containerID="439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178"
Nov 02 21:49:25.404041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.404020    4528 scope.go:122] "RemoveContainer" containerID="2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06"
Nov 02 21:49:25.412419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.412398    4528 scope.go:122] "RemoveContainer" containerID="439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178"
Nov 02 21:49:25.412663 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:49:25.412645    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178\": container with ID starting with 439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178 not found: ID does not exist" containerID="439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178"
Nov 02 21:49:25.412755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.412669    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178"} err="failed to get container status \"439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178\": rpc error: code = NotFound desc = could not find container \"439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178\": container with ID starting with 439930dc89003515bfc534665ae97401845cc78cd9639de91acf9b9130407178 not found: ID does not exist"
Nov 02 21:49:25.412755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.412686    4528 scope.go:122] "RemoveContainer" containerID="2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06"
Nov 02 21:49:25.412966 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:49:25.412945    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06\": container with ID starting with 2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06 not found: ID does not exist" containerID="2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06"
Nov 02 21:49:25.413044 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.412971    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06"} err="failed to get container status \"2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06\": rpc error: code = NotFound desc = could not find container \"2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06\": container with ID starting with 2c7fcf59d3115c092eee2ad2808117fced276dea6692637c4cc1490477b8cb06 not found: ID does not exist"
Nov 02 21:49:25.530797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.530773    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:49:25.550239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.542427    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"]
Nov 02 21:49:25.591425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.591394    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3997-openshift-default-7b7589996f-6tl7q"]
Nov 02 21:49:25.640793 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.640768    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53adbd40-c61f-444e-a2ea-905cb140f56b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53adbd40-c61f-444e-a2ea-905cb140f56b-plugins-config-volume\") pod \"53adbd40-c61f-444e-a2ea-905cb140f56b\" (UID: \"53adbd40-c61f-444e-a2ea-905cb140f56b\") "
Nov 02 21:49:25.640950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.640818    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53adbd40-c61f-444e-a2ea-905cb140f56b-kube-api-access-xw6n4\" (UniqueName: \"kubernetes.io/projected/53adbd40-c61f-444e-a2ea-905cb140f56b-kube-api-access-xw6n4\") pod \"53adbd40-c61f-444e-a2ea-905cb140f56b\" (UID: \"53adbd40-c61f-444e-a2ea-905cb140f56b\") "
Nov 02 21:49:25.641216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.641192    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53adbd40-c61f-444e-a2ea-905cb140f56b-plugins-config-volume" pod "53adbd40-c61f-444e-a2ea-905cb140f56b" (UID: "53adbd40-c61f-444e-a2ea-905cb140f56b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:49:25.642794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.642764    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53adbd40-c61f-444e-a2ea-905cb140f56b-kube-api-access-xw6n4" pod "53adbd40-c61f-444e-a2ea-905cb140f56b" (UID: "53adbd40-c61f-444e-a2ea-905cb140f56b"). InnerVolumeSpecName "kube-api-access-xw6n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:49:25.742015 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.741985    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53adbd40-c61f-444e-a2ea-905cb140f56b-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:25.742015 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:25.742022    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw6n4\" (UniqueName: \"kubernetes.io/projected/53adbd40-c61f-444e-a2ea-905cb140f56b-kube-api-access-xw6n4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:26.382449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.382414    4528 generic.go:356] "Generic (PLEG): container finished" podID="53adbd40-c61f-444e-a2ea-905cb140f56b" containerID="357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0" exitCode=0
Nov 02 21:49:26.382871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.382471    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"
Nov 02 21:49:26.382871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.382492    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5" event={"ID":"53adbd40-c61f-444e-a2ea-905cb140f56b","Type":"ContainerDied","Data":"357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0"}
Nov 02 21:49:26.382871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.382544    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5" event={"ID":"53adbd40-c61f-444e-a2ea-905cb140f56b","Type":"ContainerDied","Data":"ce00c5aa517df84311681d80c01721d9fc9f85a51428dd242d4b561f6be4239f"}
Nov 02 21:49:26.382871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.382560    4528 scope.go:122] "RemoveContainer" containerID="357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0"
Nov 02 21:49:26.391834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.391815    4528 scope.go:122] "RemoveContainer" containerID="357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0"
Nov 02 21:49:26.392128 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:49:26.392094    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0\": container with ID starting with 357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0 not found: ID does not exist" containerID="357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0"
Nov 02 21:49:26.392243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.392135    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0"} err="failed to get container status \"357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0\": rpc error: code = NotFound desc = could not find container \"357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0\": container with ID starting with 357519ca16d0bd65f93c3346de94c7fbadbcf92bafe3945f01fb834f2ed8d2a0 not found: ID does not exist"
Nov 02 21:49:26.546004 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.545972    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"]
Nov 02 21:49:26.596123 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.596086    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc0d6a-64b96bd84c-nmnt5"]
Nov 02 21:49:26.913490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.913462    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53adbd40-c61f-444e-a2ea-905cb140f56b" path="/var/lib/kubelet/pods/53adbd40-c61f-444e-a2ea-905cb140f56b/volumes"
Nov 02 21:49:26.913911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:26.913897    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" path="/var/lib/kubelet/pods/b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef/volumes"
Nov 02 21:49:35.009809 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.009777    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"]
Nov 02 21:49:35.010531 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.010514    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy"
Nov 02 21:49:35.010584 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.010535    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy"
Nov 02 21:49:35.010617 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.010584    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy"
Nov 02 21:49:35.010617 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.010593    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy"
Nov 02 21:49:35.010617 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.010612    4528 cpu_manager.go:414] "containerMap: removing container" podUID="53adbd40-c61f-444e-a2ea-905cb140f56b" containerName="payload-pre-processing"
Nov 02 21:49:35.010726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.010621    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53adbd40-c61f-444e-a2ea-905cb140f56b" containerName="payload-pre-processing"
Nov 02 21:49:35.010772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.010747    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy"
Nov 02 21:49:35.010772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.010762    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="53adbd40-c61f-444e-a2ea-905cb140f56b" containerName="payload-pre-processing"
Nov 02 21:49:35.010872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.010777    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1acbfd2-7cae-40a9-bf4f-ac7f3f8c99ef" containerName="istio-proxy"
Nov 02 21:49:35.015744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.015724    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.065195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.065134    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"]
Nov 02 21:49:35.068304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.068280    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:49:35.112655 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.112629    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:49:35.112765 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.112682    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:49:35.118193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.118176    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:49:35.159600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.159574    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42cg9\" (UniqueName: \"kubernetes.io/projected/59da25a0-08cf-4900-9c8e-c124c839a922-kube-api-access-42cg9\") pod \"maas-api-revoke-sub-0701606c-bbk2g\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.159754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.159734    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0701606c-bbk2g\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.159807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.159776    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0701606c-bbk2g\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.260708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.260679    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cg9\" (UniqueName: \"kubernetes.io/projected/59da25a0-08cf-4900-9c8e-c124c839a922-kube-api-access-42cg9\") pod \"maas-api-revoke-sub-0701606c-bbk2g\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.260880 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.260825    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0701606c-bbk2g\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.260880 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.260867    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0701606c-bbk2g\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.261466 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.261437    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0701606c-bbk2g\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.263090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.263070    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0701606c-bbk2g\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.317105 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.317035    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cg9\" (UniqueName: \"kubernetes.io/projected/59da25a0-08cf-4900-9c8e-c124c839a922-kube-api-access-42cg9\") pod \"maas-api-revoke-sub-0701606c-bbk2g\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.328001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.327981    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:35.693932 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:49:35.693902    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59da25a0_08cf_4900_9c8e_c124c839a922.slice/crio-59352584d6b1d8be5e228e17fcd25baada0db0a05dd4d472291c7069b0c9a483 WatchSource:0}: Error finding container 59352584d6b1d8be5e228e17fcd25baada0db0a05dd4d472291c7069b0c9a483: Status 404 returned error can't find the container with id 59352584d6b1d8be5e228e17fcd25baada0db0a05dd4d472291c7069b0c9a483
Nov 02 21:49:35.696941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:35.696916    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"]
Nov 02 21:49:36.445366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:36.445282    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g" event={"ID":"59da25a0-08cf-4900-9c8e-c124c839a922","Type":"ContainerStarted","Data":"dc561ecb6132b11a6d7bb87c168b6e4eeae66f110cfe210cd775cbbe545a3cd0"}
Nov 02 21:49:36.445366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:36.445339    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g" event={"ID":"59da25a0-08cf-4900-9c8e-c124c839a922","Type":"ContainerStarted","Data":"59352584d6b1d8be5e228e17fcd25baada0db0a05dd4d472291c7069b0c9a483"}
Nov 02 21:49:36.560573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:36.560517    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g" podStartSLOduration=2.560500085 podStartE2EDuration="2.560500085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:36.557805609 +0000 UTC m=+2463.821341926" watchObservedRunningTime="2026-11-02 21:49:36.560500085 +0000 UTC m=+2463.824036403"
Nov 02 21:49:37.451713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:37.451677    4528 generic.go:356] "Generic (PLEG): container finished" podID="59da25a0-08cf-4900-9c8e-c124c839a922" containerID="dc561ecb6132b11a6d7bb87c168b6e4eeae66f110cfe210cd775cbbe545a3cd0" exitCode=0
Nov 02 21:49:37.452091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:37.451762    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g" event={"ID":"59da25a0-08cf-4900-9c8e-c124c839a922","Type":"ContainerDied","Data":"dc561ecb6132b11a6d7bb87c168b6e4eeae66f110cfe210cd775cbbe545a3cd0"}
Nov 02 21:49:38.654470 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.654447    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:38.694743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.694714    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-xks-ca\") pod \"59da25a0-08cf-4900-9c8e-c124c839a922\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") "
Nov 02 21:49:38.694907 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.694760    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-openshift-ca\") pod \"59da25a0-08cf-4900-9c8e-c124c839a922\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") "
Nov 02 21:49:38.694907 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.694899    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59da25a0-08cf-4900-9c8e-c124c839a922-kube-api-access-42cg9\" (UniqueName: \"kubernetes.io/projected/59da25a0-08cf-4900-9c8e-c124c839a922-kube-api-access-42cg9\") pod \"59da25a0-08cf-4900-9c8e-c124c839a922\" (UID: \"59da25a0-08cf-4900-9c8e-c124c839a922\") "
Nov 02 21:49:38.695241 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.695210    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-openshift-ca" pod "59da25a0-08cf-4900-9c8e-c124c839a922" (UID: "59da25a0-08cf-4900-9c8e-c124c839a922"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:49:38.695369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.695322    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:38.697008 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.696977    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-xks-ca" pod "59da25a0-08cf-4900-9c8e-c124c839a922" (UID: "59da25a0-08cf-4900-9c8e-c124c839a922"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:49:38.697189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.697169    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59da25a0-08cf-4900-9c8e-c124c839a922-kube-api-access-42cg9" pod "59da25a0-08cf-4900-9c8e-c124c839a922" (UID: "59da25a0-08cf-4900-9c8e-c124c839a922"). InnerVolumeSpecName "kube-api-access-42cg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:49:38.714364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.709398    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"]
Nov 02 21:49:38.714364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.710688    4528 cpu_manager.go:414] "containerMap: removing container" podUID="59da25a0-08cf-4900-9c8e-c124c839a922" containerName="revoke-keys"
Nov 02 21:49:38.714364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.710706    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59da25a0-08cf-4900-9c8e-c124c839a922" containerName="revoke-keys"
Nov 02 21:49:38.714364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.710917    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="59da25a0-08cf-4900-9c8e-c124c839a922" containerName="revoke-keys"
Nov 02 21:49:38.715995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.715977    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:38.779766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.779733    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-eabda2\""
Nov 02 21:49:38.782465 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.780263    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-eabda2-dockercfg-hlq5p\""
Nov 02 21:49:38.787070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.784976    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"]
Nov 02 21:49:38.817002 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.816948    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31213d06-22c3-430f-a99e-a54d475d1597-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw\" (UID: \"31213d06-22c3-430f-a99e-a54d475d1597\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:38.817529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.817492    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9p2v\" (UniqueName: \"kubernetes.io/projected/31213d06-22c3-430f-a99e-a54d475d1597-kube-api-access-x9p2v\") pod \"payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw\" (UID: \"31213d06-22c3-430f-a99e-a54d475d1597\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:38.831283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.831248    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42cg9\" (UniqueName: \"kubernetes.io/projected/59da25a0-08cf-4900-9c8e-c124c839a922-kube-api-access-42cg9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:38.831283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.831288    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da25a0-08cf-4900-9c8e-c124c839a922-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:38.861360 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.861329    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"]
Nov 02 21:49:38.865255 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.865234    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:38.917558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.917530    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"]
Nov 02 21:49:38.932373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.932347    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7c5\" (UniqueName: \"kubernetes.io/projected/8ce62acb-45af-4da8-a5ae-fa6d381144a6-kube-api-access-qh7c5\") pod \"payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms\" (UID: \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:38.932510 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.932411    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9p2v\" (UniqueName: \"kubernetes.io/projected/31213d06-22c3-430f-a99e-a54d475d1597-kube-api-access-x9p2v\") pod \"payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw\" (UID: \"31213d06-22c3-430f-a99e-a54d475d1597\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:38.932561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.932516    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce62acb-45af-4da8-a5ae-fa6d381144a6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms\" (UID: \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:38.932657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.932646    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31213d06-22c3-430f-a99e-a54d475d1597-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw\" (UID: \"31213d06-22c3-430f-a99e-a54d475d1597\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:38.933335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.933311    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31213d06-22c3-430f-a99e-a54d475d1597-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw\" (UID: \"31213d06-22c3-430f-a99e-a54d475d1597\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:38.989016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:38.988945    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9p2v\" (UniqueName: \"kubernetes.io/projected/31213d06-22c3-430f-a99e-a54d475d1597-kube-api-access-x9p2v\") pod \"payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw\" (UID: \"31213d06-22c3-430f-a99e-a54d475d1597\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:39.029826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.029794    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:39.033487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.033460    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7c5\" (UniqueName: \"kubernetes.io/projected/8ce62acb-45af-4da8-a5ae-fa6d381144a6-kube-api-access-qh7c5\") pod \"payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms\" (UID: \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:39.033583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.033554    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce62acb-45af-4da8-a5ae-fa6d381144a6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms\" (UID: \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:39.034270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.034250    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce62acb-45af-4da8-a5ae-fa6d381144a6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms\" (UID: \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:39.148131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.140928    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7c5\" (UniqueName: \"kubernetes.io/projected/8ce62acb-45af-4da8-a5ae-fa6d381144a6-kube-api-access-qh7c5\") pod \"payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms\" (UID: \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:39.182602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.182558    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:39.378085 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.378050    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:49:39.403943 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:49:39.403909    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31213d06_22c3_430f_a99e_a54d475d1597.slice/crio-b92af4e322494d0a2ad9da0f54565ced5a36881d5cda2e3a53f0e7a7b7eaea61 WatchSource:0}: Error finding container b92af4e322494d0a2ad9da0f54565ced5a36881d5cda2e3a53f0e7a7b7eaea61: Status 404 returned error can't find the container with id b92af4e322494d0a2ad9da0f54565ced5a36881d5cda2e3a53f0e7a7b7eaea61
Nov 02 21:49:39.405219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.405193    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"]
Nov 02 21:49:39.474583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.474549    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw" event={"ID":"31213d06-22c3-430f-a99e-a54d475d1597","Type":"ContainerStarted","Data":"b92af4e322494d0a2ad9da0f54565ced5a36881d5cda2e3a53f0e7a7b7eaea61"}
Nov 02 21:49:39.476287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.476261    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g" event={"ID":"59da25a0-08cf-4900-9c8e-c124c839a922","Type":"ContainerDied","Data":"59352584d6b1d8be5e228e17fcd25baada0db0a05dd4d472291c7069b0c9a483"}
Nov 02 21:49:39.476398 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.476312    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59352584d6b1d8be5e228e17fcd25baada0db0a05dd4d472291c7069b0c9a483"
Nov 02 21:49:39.476445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.476400    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"
Nov 02 21:49:39.587717 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:49:39.587683    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce62acb_45af_4da8_a5ae_fa6d381144a6.slice/crio-b0ebcb6d7e3038d8ff9230674ce8314073fca5d2c7f5f7b5c0cfb529524f6685 WatchSource:0}: Error finding container b0ebcb6d7e3038d8ff9230674ce8314073fca5d2c7f5f7b5c0cfb529524f6685: Status 404 returned error can't find the container with id b0ebcb6d7e3038d8ff9230674ce8314073fca5d2c7f5f7b5c0cfb529524f6685
Nov 02 21:49:39.590094 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:39.590070    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"]
Nov 02 21:49:40.483269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:40.483238    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms" event={"ID":"8ce62acb-45af-4da8-a5ae-fa6d381144a6","Type":"ContainerStarted","Data":"795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2"}
Nov 02 21:49:40.483768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:40.483286    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms" event={"ID":"8ce62acb-45af-4da8-a5ae-fa6d381144a6","Type":"ContainerStarted","Data":"b0ebcb6d7e3038d8ff9230674ce8314073fca5d2c7f5f7b5c0cfb529524f6685"}
Nov 02 21:49:40.483768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:40.483334    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:40.484987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:40.484964    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw" event={"ID":"31213d06-22c3-430f-a99e-a54d475d1597","Type":"ContainerStarted","Data":"d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719"}
Nov 02 21:49:40.485097 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:40.485070    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:40.597653 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:40.597608    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms" podStartSLOduration=2.597595827 podStartE2EDuration="2.597595827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:40.596750477 +0000 UTC m=+2467.860286793" watchObservedRunningTime="2026-11-02 21:49:40.597595827 +0000 UTC m=+2467.861132144"
Nov 02 21:49:44.111889 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:49:44.111851    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:49:51.493836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:51.493802    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:49:51.494371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:51.493866    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:49:51.610665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:51.610608    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw" podStartSLOduration=13.610592788 podStartE2EDuration="13.610592788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:40.724388897 +0000 UTC m=+2467.987925214" watchObservedRunningTime="2026-11-02 21:49:51.610592788 +0000 UTC m=+2478.874129106"
Nov 02 21:49:53.063975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.063942    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"]
Nov 02 21:49:53.068070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.068054    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.119395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.119364    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"]
Nov 02 21:49:53.130596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.130555    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:49:53.130596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.130556    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:49:53.166850 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.166823    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:49:53.166985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.166910    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:49:53.199281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.199250    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp97f\" (UniqueName: \"kubernetes.io/projected/42a2d9c9-eba9-48bb-bc4a-4519424265ad-kube-api-access-sp97f\") pod \"maas-api-revoke-sub-63e1b4aa-g7sfs\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.199395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.199359    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e1b4aa-g7sfs\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.199450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.199396    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e1b4aa-g7sfs\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.300312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.300279    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e1b4aa-g7sfs\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.300483 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.300330    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e1b4aa-g7sfs\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.300552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.300477    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp97f\" (UniqueName: \"kubernetes.io/projected/42a2d9c9-eba9-48bb-bc4a-4519424265ad-kube-api-access-sp97f\") pod \"maas-api-revoke-sub-63e1b4aa-g7sfs\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.300870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.300849    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e1b4aa-g7sfs\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.302639 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.302616    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e1b4aa-g7sfs\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.362768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.362713    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp97f\" (UniqueName: \"kubernetes.io/projected/42a2d9c9-eba9-48bb-bc4a-4519424265ad-kube-api-access-sp97f\") pod \"maas-api-revoke-sub-63e1b4aa-g7sfs\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.382064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.382046    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:53.751844 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:49:53.751801    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a2d9c9_eba9_48bb_bc4a_4519424265ad.slice/crio-5743083584f87d1ded7fb7cda077fd2a8634d1d388b72a9cc3534d63b91f9753 WatchSource:0}: Error finding container 5743083584f87d1ded7fb7cda077fd2a8634d1d388b72a9cc3534d63b91f9753: Status 404 returned error can't find the container with id 5743083584f87d1ded7fb7cda077fd2a8634d1d388b72a9cc3534d63b91f9753
Nov 02 21:49:53.754824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:53.754714    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"]
Nov 02 21:49:54.566644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:54.566614    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs" event={"ID":"42a2d9c9-eba9-48bb-bc4a-4519424265ad","Type":"ContainerStarted","Data":"9334722ba3c961c1c344d86d6acf6ff7fe2c08e9a0ca10418b7524c6712f9022"}
Nov 02 21:49:54.567039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:54.566668    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs" event={"ID":"42a2d9c9-eba9-48bb-bc4a-4519424265ad","Type":"ContainerStarted","Data":"5743083584f87d1ded7fb7cda077fd2a8634d1d388b72a9cc3534d63b91f9753"}
Nov 02 21:49:54.681091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:54.681041    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs" podStartSLOduration=1.6810262630000001 podStartE2EDuration="1.681026263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:54.679637016 +0000 UTC m=+2481.943173334" watchObservedRunningTime="2026-11-02 21:49:54.681026263 +0000 UTC m=+2481.944562578"
Nov 02 21:49:55.573354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:55.573319    4528 generic.go:356] "Generic (PLEG): container finished" podID="42a2d9c9-eba9-48bb-bc4a-4519424265ad" containerID="9334722ba3c961c1c344d86d6acf6ff7fe2c08e9a0ca10418b7524c6712f9022" exitCode=0
Nov 02 21:49:55.573742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:55.573399    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs" event={"ID":"42a2d9c9-eba9-48bb-bc4a-4519424265ad","Type":"ContainerDied","Data":"9334722ba3c961c1c344d86d6acf6ff7fe2c08e9a0ca10418b7524c6712f9022"}
Nov 02 21:49:56.730044 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.730016    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:56.845532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.845459    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-openshift-ca\") pod \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") "
Nov 02 21:49:56.845669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.845574    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-xks-ca\") pod \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") "
Nov 02 21:49:56.845721 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.845669    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a2d9c9-eba9-48bb-bc4a-4519424265ad-kube-api-access-sp97f\" (UniqueName: \"kubernetes.io/projected/42a2d9c9-eba9-48bb-bc4a-4519424265ad-kube-api-access-sp97f\") pod \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\" (UID: \"42a2d9c9-eba9-48bb-bc4a-4519424265ad\") "
Nov 02 21:49:56.845901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.845874    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-openshift-ca" pod "42a2d9c9-eba9-48bb-bc4a-4519424265ad" (UID: "42a2d9c9-eba9-48bb-bc4a-4519424265ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:49:56.847846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.847820    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-xks-ca" pod "42a2d9c9-eba9-48bb-bc4a-4519424265ad" (UID: "42a2d9c9-eba9-48bb-bc4a-4519424265ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:49:56.848050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.848031    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a2d9c9-eba9-48bb-bc4a-4519424265ad-kube-api-access-sp97f" pod "42a2d9c9-eba9-48bb-bc4a-4519424265ad" (UID: "42a2d9c9-eba9-48bb-bc4a-4519424265ad"). InnerVolumeSpecName "kube-api-access-sp97f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:49:56.946734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.946702    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp97f\" (UniqueName: \"kubernetes.io/projected/42a2d9c9-eba9-48bb-bc4a-4519424265ad-kube-api-access-sp97f\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:56.946734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.946733    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:56.946908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:56.946748    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a2d9c9-eba9-48bb-bc4a-4519424265ad-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:49:57.585833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:57.585795    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs" event={"ID":"42a2d9c9-eba9-48bb-bc4a-4519424265ad","Type":"ContainerDied","Data":"5743083584f87d1ded7fb7cda077fd2a8634d1d388b72a9cc3534d63b91f9753"}
Nov 02 21:49:57.586043 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:57.585839    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"
Nov 02 21:49:57.586043 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:49:57.585869    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5743083584f87d1ded7fb7cda077fd2a8634d1d388b72a9cc3534d63b91f9753"
Nov 02 21:50:01.095979 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:01.095945    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"]
Nov 02 21:50:01.096457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:01.096184    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy" containerID="cri-o://7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c" gracePeriod=30
Nov 02 21:50:06.359990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.359966    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:50:06.452626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.452548    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-token\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-token\") pod \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") "
Nov 02 21:50:06.452626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.452601    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istiod-ca-cert\") pod \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") "
Nov 02 21:50:06.452807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.452647    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-envoy\") pod \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") "
Nov 02 21:50:06.452807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.452709    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-data\") pod \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") "
Nov 02 21:50:06.452807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.452738    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-socket\") pod \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") "
Nov 02 21:50:06.452807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.452783    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-certs\") pod \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") "
Nov 02 21:50:06.453019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.452821    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-kube-api-access-kfbtm\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-kube-api-access-kfbtm\") pod \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") "
Nov 02 21:50:06.453019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.452886    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-credential-socket\") pod \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") "
Nov 02 21:50:06.453019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.452953    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-podinfo\") pod \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\" (UID: \"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9\") "
Nov 02 21:50:06.453282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.453185    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istiod-ca-cert" pod "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" (UID: "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:06.453282 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.453242    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-certs" pod "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" (UID: "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:06.453455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.453417    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-data" pod "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" (UID: "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:06.453455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.453446    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-credential-socket" pod "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" (UID: "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:06.453600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.453582    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-socket" pod "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" (UID: "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:06.453654 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.453635    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:06.453705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.453657    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:06.453705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.453674    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:06.453705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.453689    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:06.454901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.454878    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-envoy" pod "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" (UID: "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:06.455172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.455150    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-kube-api-access-kfbtm" pod "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" (UID: "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9"). InnerVolumeSpecName "kube-api-access-kfbtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:06.455262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.455241    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-token" pod "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" (UID: "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:06.455360 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.455342    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-podinfo" pod "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" (UID: "1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:50:06.554519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.554483    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:06.554519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.554508    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbtm\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-kube-api-access-kfbtm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:06.554519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.554518    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:06.554519 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.554527    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:06.554887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.554535    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:06.644961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.644925    4528 generic.go:356] "Generic (PLEG): container finished" podID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerID="7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c" exitCode=0
Nov 02 21:50:06.645162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.644976    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"
Nov 02 21:50:06.645162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.645009    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" event={"ID":"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9","Type":"ContainerDied","Data":"7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c"}
Nov 02 21:50:06.645162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.645096    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff" event={"ID":"1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9","Type":"ContainerDied","Data":"d6413ae2316f908e41fe774718ab5b41f6d2403da79c36baa498ed92c432154e"}
Nov 02 21:50:06.645162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.645135    4528 scope.go:122] "RemoveContainer" containerID="7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c"
Nov 02 21:50:06.654876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.654859    4528 scope.go:122] "RemoveContainer" containerID="b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf"
Nov 02 21:50:06.662840 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.662824    4528 scope.go:122] "RemoveContainer" containerID="7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c"
Nov 02 21:50:06.663087 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:50:06.663067    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c\": container with ID starting with 7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c not found: ID does not exist" containerID="7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c"
Nov 02 21:50:06.663202 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.663095    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c"} err="failed to get container status \"7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c\": rpc error: code = NotFound desc = could not find container \"7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c\": container with ID starting with 7461f5b4d0807339701dce15a0e18263573cfef25be5e22b46da8b3fe461c74c not found: ID does not exist"
Nov 02 21:50:06.663202 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.663130    4528 scope.go:122] "RemoveContainer" containerID="b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf"
Nov 02 21:50:06.663402 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:50:06.663384    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf\": container with ID starting with b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf not found: ID does not exist" containerID="b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf"
Nov 02 21:50:06.663449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.663409    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf"} err="failed to get container status \"b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf\": rpc error: code = NotFound desc = could not find container \"b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf\": container with ID starting with b194a6cefdb5150a58d837f187f14b67a186f997beeb8b168af14e7b9430eacf not found: ID does not exist"
Nov 02 21:50:06.810701 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.810664    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"]
Nov 02 21:50:06.813774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.813753    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-mllff"]
Nov 02 21:50:06.913779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:06.913738    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" path="/var/lib/kubelet/pods/1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9/volumes"
Nov 02 21:50:08.494330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:08.494298    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"]
Nov 02 21:50:08.494756 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:08.494495    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy" containerID="cri-o://8709fbd4be1ba741a9300799e04cbd791397f9718ba432c386de471832e68365" gracePeriod=30
Nov 02 21:50:08.545961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:08.545927    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"]
Nov 02 21:50:08.546260 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:08.546201    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy" containerID="cri-o://e6f4fefafe3102b7ae9fcbe2024f778377e1b6f17fc9a57b693fa0247ea36ab8" gracePeriod=30
Nov 02 21:50:09.247563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.247525    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"]
Nov 02 21:50:09.248090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.248075    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy"
Nov 02 21:50:09.248161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.248094    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy"
Nov 02 21:50:09.248161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.248131    4528 cpu_manager.go:414] "containerMap: removing container" podUID="42a2d9c9-eba9-48bb-bc4a-4519424265ad" containerName="revoke-keys"
Nov 02 21:50:09.248161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.248136    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a2d9c9-eba9-48bb-bc4a-4519424265ad" containerName="revoke-keys"
Nov 02 21:50:09.248161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.248155    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy"
Nov 02 21:50:09.248161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.248161    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy"
Nov 02 21:50:09.248317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.248258    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a2d9c9-eba9-48bb-bc4a-4519424265ad" containerName="revoke-keys"
Nov 02 21:50:09.248317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.248270    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy"
Nov 02 21:50:09.248317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.248278    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af8bacd-5cbc-4c50-97a4-0fdf084d4bd9" containerName="istio-proxy"
Nov 02 21:50:09.253005 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.252987    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.283596 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.283567    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbb52\" (UniqueName: \"kubernetes.io/projected/e473c7b6-57d8-4610-a0f0-87d759da005e-kube-api-access-wbb52\") pod \"maas-api-revoke-sub-9c4c8948-t54sz\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.283718 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.283705    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c4c8948-t54sz\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.283766 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.283741    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c4c8948-t54sz\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.304027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.304001    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"]
Nov 02 21:50:09.306877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.306855    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:50:09.351496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.351472    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:50:09.351496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.351471    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:50:09.351686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.351500    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:50:09.385367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.385340    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c4c8948-t54sz\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.385497 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.385378    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c4c8948-t54sz\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.385497 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.385435    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbb52\" (UniqueName: \"kubernetes.io/projected/e473c7b6-57d8-4610-a0f0-87d759da005e-kube-api-access-wbb52\") pod \"maas-api-revoke-sub-9c4c8948-t54sz\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.386026 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.386003    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c4c8948-t54sz\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.387595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.387573    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c4c8948-t54sz\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.443493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.443463    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbb52\" (UniqueName: \"kubernetes.io/projected/e473c7b6-57d8-4610-a0f0-87d759da005e-kube-api-access-wbb52\") pod \"maas-api-revoke-sub-9c4c8948-t54sz\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.566039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.566003    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:09.935854 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:50:09.935827    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode473c7b6_57d8_4610_a0f0_87d759da005e.slice/crio-5d1ba63628850b9c09ffb466d30620c6cce667b5b4b17abb8bdbe80ec0f44075 WatchSource:0}: Error finding container 5d1ba63628850b9c09ffb466d30620c6cce667b5b4b17abb8bdbe80ec0f44075: Status 404 returned error can't find the container with id 5d1ba63628850b9c09ffb466d30620c6cce667b5b4b17abb8bdbe80ec0f44075
Nov 02 21:50:09.938002 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:09.937976    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"]
Nov 02 21:50:10.668618 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:10.668581    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz" event={"ID":"e473c7b6-57d8-4610-a0f0-87d759da005e","Type":"ContainerStarted","Data":"4117ebfca3e21bcb05a7d8d929d45504c2c082ad2e199c38a4847e7e4acc1161"}
Nov 02 21:50:10.668980 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:10.668640    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz" event={"ID":"e473c7b6-57d8-4610-a0f0-87d759da005e","Type":"ContainerStarted","Data":"5d1ba63628850b9c09ffb466d30620c6cce667b5b4b17abb8bdbe80ec0f44075"}
Nov 02 21:50:10.783340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:10.783297    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz" podStartSLOduration=1.783282641 podStartE2EDuration="1.783282641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:10.781326101 +0000 UTC m=+2498.044862416" watchObservedRunningTime="2026-11-02 21:50:10.783282641 +0000 UTC m=+2498.046818957"
Nov 02 21:50:11.675538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:11.675503    4528 generic.go:356] "Generic (PLEG): container finished" podID="e473c7b6-57d8-4610-a0f0-87d759da005e" containerID="4117ebfca3e21bcb05a7d8d929d45504c2c082ad2e199c38a4847e7e4acc1161" exitCode=0
Nov 02 21:50:11.675944 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:11.675570    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz" event={"ID":"e473c7b6-57d8-4610-a0f0-87d759da005e","Type":"ContainerDied","Data":"4117ebfca3e21bcb05a7d8d929d45504c2c082ad2e199c38a4847e7e4acc1161"}
Nov 02 21:50:12.813170 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:12.813143    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:12.923017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:12.922860    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-openshift-ca\") pod \"e473c7b6-57d8-4610-a0f0-87d759da005e\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") "
Nov 02 21:50:12.923017 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:12.922995    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-xks-ca\") pod \"e473c7b6-57d8-4610-a0f0-87d759da005e\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") "
Nov 02 21:50:12.923405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:12.923087    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e473c7b6-57d8-4610-a0f0-87d759da005e-kube-api-access-wbb52\" (UniqueName: \"kubernetes.io/projected/e473c7b6-57d8-4610-a0f0-87d759da005e-kube-api-access-wbb52\") pod \"e473c7b6-57d8-4610-a0f0-87d759da005e\" (UID: \"e473c7b6-57d8-4610-a0f0-87d759da005e\") "
Nov 02 21:50:12.923405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:12.923382    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-openshift-ca" pod "e473c7b6-57d8-4610-a0f0-87d759da005e" (UID: "e473c7b6-57d8-4610-a0f0-87d759da005e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:12.923754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:12.923580    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:12.925614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:12.925586    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-xks-ca" pod "e473c7b6-57d8-4610-a0f0-87d759da005e" (UID: "e473c7b6-57d8-4610-a0f0-87d759da005e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:50:12.925773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:12.925751    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e473c7b6-57d8-4610-a0f0-87d759da005e-kube-api-access-wbb52" pod "e473c7b6-57d8-4610-a0f0-87d759da005e" (UID: "e473c7b6-57d8-4610-a0f0-87d759da005e"). InnerVolumeSpecName "kube-api-access-wbb52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:13.025269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.025228    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e473c7b6-57d8-4610-a0f0-87d759da005e-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.025269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.025262    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbb52\" (UniqueName: \"kubernetes.io/projected/e473c7b6-57d8-4610-a0f0-87d759da005e-kube-api-access-wbb52\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.691294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.691260    4528 generic.go:356] "Generic (PLEG): container finished" podID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerID="8709fbd4be1ba741a9300799e04cbd791397f9718ba432c386de471832e68365" exitCode=0
Nov 02 21:50:13.691461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.691340    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" event={"ID":"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111","Type":"ContainerDied","Data":"8709fbd4be1ba741a9300799e04cbd791397f9718ba432c386de471832e68365"}
Nov 02 21:50:13.692801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.692777    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz" event={"ID":"e473c7b6-57d8-4610-a0f0-87d759da005e","Type":"ContainerDied","Data":"5d1ba63628850b9c09ffb466d30620c6cce667b5b4b17abb8bdbe80ec0f44075"}
Nov 02 21:50:13.692922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.692812    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1ba63628850b9c09ffb466d30620c6cce667b5b4b17abb8bdbe80ec0f44075"
Nov 02 21:50:13.692922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.692866    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"
Nov 02 21:50:13.695197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.695168    4528 generic.go:356] "Generic (PLEG): container finished" podID="573adb50-82ab-442f-a850-ce34862f2b10" containerID="e6f4fefafe3102b7ae9fcbe2024f778377e1b6f17fc9a57b693fa0247ea36ab8" exitCode=0
Nov 02 21:50:13.695308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.695239    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" event={"ID":"573adb50-82ab-442f-a850-ce34862f2b10","Type":"ContainerDied","Data":"e6f4fefafe3102b7ae9fcbe2024f778377e1b6f17fc9a57b693fa0247ea36ab8"}
Nov 02 21:50:13.779375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.779353    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:50:13.834583 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.834551    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-socket\") pod \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") "
Nov 02 21:50:13.835021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.834659    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-podinfo\") pod \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") "
Nov 02 21:50:13.835021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.834691    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istiod-ca-cert\") pod \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") "
Nov 02 21:50:13.835021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.834722    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-certs\") pod \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") "
Nov 02 21:50:13.835021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.834755    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-credential-socket\") pod \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") "
Nov 02 21:50:13.835021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.834859    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-envoy\") pod \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") "
Nov 02 21:50:13.835021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.834903    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-data\") pod \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") "
Nov 02 21:50:13.835021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.834988    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-kube-api-access-jr746\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-kube-api-access-jr746\") pod \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") "
Nov 02 21:50:13.835443 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.835066    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-token\") pod \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\" (UID: \"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111\") "
Nov 02 21:50:13.836080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.836055    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-certs" pod "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" (UID: "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.837096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.837073    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istiod-ca-cert" pod "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" (UID: "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:13.837715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.837693    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-socket" pod "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" (UID: "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.837852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.837781    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-data" pod "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" (UID: "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.838372 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.838351    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-credential-socket" pod "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" (UID: "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.839939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.839914    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-token" pod "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" (UID: "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:13.842490 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.842468    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-kube-api-access-jr746" pod "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" (UID: "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111"). InnerVolumeSpecName "kube-api-access-jr746". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:13.842579 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.842469    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-podinfo" pod "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" (UID: "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:50:13.842696 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.842674    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-envoy" pod "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" (UID: "e3bc3d67-6e03-4bbb-8316-4f28a4a6b111"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.875898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.875871    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:50:13.935880 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.935850    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-istio-token\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-istio-token\") pod \"573adb50-82ab-442f-a850-ce34862f2b10\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") "
Nov 02 21:50:13.936087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.935904    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-kube-api-access-7j4sn\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-kube-api-access-7j4sn\") pod \"573adb50-82ab-442f-a850-ce34862f2b10\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") "
Nov 02 21:50:13.936087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.935938    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-credential-socket\") pod \"573adb50-82ab-442f-a850-ce34862f2b10\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") "
Nov 02 21:50:13.936087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.935972    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-certs\") pod \"573adb50-82ab-442f-a850-ce34862f2b10\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") "
Nov 02 21:50:13.936087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936059    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/573adb50-82ab-442f-a850-ce34862f2b10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573adb50-82ab-442f-a850-ce34862f2b10-istiod-ca-cert\") pod \"573adb50-82ab-442f-a850-ce34862f2b10\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") "
Nov 02 21:50:13.936315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936096    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-data\") pod \"573adb50-82ab-442f-a850-ce34862f2b10\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") "
Nov 02 21:50:13.936315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936144    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-socket\") pod \"573adb50-82ab-442f-a850-ce34862f2b10\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") "
Nov 02 21:50:13.936315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936204    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/573adb50-82ab-442f-a850-ce34862f2b10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573adb50-82ab-442f-a850-ce34862f2b10-istio-podinfo\") pod \"573adb50-82ab-442f-a850-ce34862f2b10\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") "
Nov 02 21:50:13.936315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936234    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-envoy\") pod \"573adb50-82ab-442f-a850-ce34862f2b10\" (UID: \"573adb50-82ab-442f-a850-ce34862f2b10\") "
Nov 02 21:50:13.936533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936295    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-credential-socket" pod "573adb50-82ab-442f-a850-ce34862f2b10" (UID: "573adb50-82ab-442f-a850-ce34862f2b10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.936533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936440    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573adb50-82ab-442f-a850-ce34862f2b10-istiod-ca-cert" pod "573adb50-82ab-442f-a850-ce34862f2b10" (UID: "573adb50-82ab-442f-a850-ce34862f2b10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:13.936649 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936608    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-socket" pod "573adb50-82ab-442f-a850-ce34862f2b10" (UID: "573adb50-82ab-442f-a850-ce34862f2b10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.936803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936778    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573adb50-82ab-442f-a850-ce34862f2b10-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.936803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936805    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936818    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936830    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936843    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr746\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-kube-api-access-jr746\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936857    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936869    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936881    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936893    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936913    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936925    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937014 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.936938    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:13.937580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.937344    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-certs" pod "573adb50-82ab-442f-a850-ce34862f2b10" (UID: "573adb50-82ab-442f-a850-ce34862f2b10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.937635 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.937578    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-data" pod "573adb50-82ab-442f-a850-ce34862f2b10" (UID: "573adb50-82ab-442f-a850-ce34862f2b10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.938413 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.938387    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-istio-token" pod "573adb50-82ab-442f-a850-ce34862f2b10" (UID: "573adb50-82ab-442f-a850-ce34862f2b10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:13.938791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.938769    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/573adb50-82ab-442f-a850-ce34862f2b10-istio-podinfo" pod "573adb50-82ab-442f-a850-ce34862f2b10" (UID: "573adb50-82ab-442f-a850-ce34862f2b10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:50:13.938894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.938808    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-envoy" pod "573adb50-82ab-442f-a850-ce34862f2b10" (UID: "573adb50-82ab-442f-a850-ce34862f2b10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:13.939069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:13.939050    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-kube-api-access-7j4sn" pod "573adb50-82ab-442f-a850-ce34862f2b10" (UID: "573adb50-82ab-442f-a850-ce34862f2b10"). InnerVolumeSpecName "kube-api-access-7j4sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:14.038471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.038390    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:14.038471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.038429    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j4sn\" (UniqueName: \"kubernetes.io/projected/573adb50-82ab-442f-a850-ce34862f2b10-kube-api-access-7j4sn\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:14.038471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.038444    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:14.038471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.038457    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:14.038471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.038471    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573adb50-82ab-442f-a850-ce34862f2b10-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:14.038750 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.038484    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573adb50-82ab-442f-a850-ce34862f2b10-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:14.112654 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:50:14.112617    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:50:14.701565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.701540    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"
Nov 02 21:50:14.701565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.701549    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs" event={"ID":"573adb50-82ab-442f-a850-ce34862f2b10","Type":"ContainerDied","Data":"68f464b15c4bd9b32683c91b21931d3a6875e790b5274f0a2a82b3da677d25aa"}
Nov 02 21:50:14.701833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.701622    4528 scope.go:122] "RemoveContainer" containerID="e6f4fefafe3102b7ae9fcbe2024f778377e1b6f17fc9a57b693fa0247ea36ab8"
Nov 02 21:50:14.703087 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.703068    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"
Nov 02 21:50:14.703211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.703090    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw" event={"ID":"e3bc3d67-6e03-4bbb-8316-4f28a4a6b111","Type":"ContainerDied","Data":"7d5925220c40f1f5da5d2f16239d514dd6beae4334ef605c3d507af4fd2644f3"}
Nov 02 21:50:14.712254 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.712174    4528 scope.go:122] "RemoveContainer" containerID="8709fbd4be1ba741a9300799e04cbd791397f9718ba432c386de471832e68365"
Nov 02 21:50:14.867365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.867330    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"]
Nov 02 21:50:14.917283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.917257    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b65pw"]
Nov 02 21:50:14.979947 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.979879    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"]
Nov 02 21:50:14.986019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:14.985998    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-n8hjs"]
Nov 02 21:50:16.912518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:16.912487    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573adb50-82ab-442f-a850-ce34862f2b10" path="/var/lib/kubelet/pods/573adb50-82ab-442f-a850-ce34862f2b10/volumes"
Nov 02 21:50:16.912945 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:16.912931    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" path="/var/lib/kubelet/pods/e3bc3d67-6e03-4bbb-8316-4f28a4a6b111/volumes"
Nov 02 21:50:19.914174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:19.914142    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"]
Nov 02 21:50:20.019847 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:20.019818    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"]
Nov 02 21:50:24.915199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:24.915143    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x" podUID="9b970845-70c6-47e3-a1d6-f5d2527877f1" containerName="payload-processing" containerID="cri-o://e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226" gracePeriod=25
Nov 02 21:50:25.020200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.020160    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k" podUID="6644f753-dd82-4c10-8775-72ce9e609997" containerName="payload-pre-processing" containerID="cri-o://e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78" gracePeriod=25
Nov 02 21:50:25.053641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.053609    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:50:25.131801 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.131770    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"]
Nov 02 21:50:25.132486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.132463    4528 cpu_manager.go:414] "containerMap: removing container" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy"
Nov 02 21:50:25.132486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.132485    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy"
Nov 02 21:50:25.132681 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.132531    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy"
Nov 02 21:50:25.132681 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.132539    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy"
Nov 02 21:50:25.132681 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.132551    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e473c7b6-57d8-4610-a0f0-87d759da005e" containerName="revoke-keys"
Nov 02 21:50:25.132681 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.132559    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e473c7b6-57d8-4610-a0f0-87d759da005e" containerName="revoke-keys"
Nov 02 21:50:25.132878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.132691    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3bc3d67-6e03-4bbb-8316-4f28a4a6b111" containerName="istio-proxy"
Nov 02 21:50:25.132878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.132707    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e473c7b6-57d8-4610-a0f0-87d759da005e" containerName="revoke-keys"
Nov 02 21:50:25.132878 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.132722    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="573adb50-82ab-442f-a850-ce34862f2b10" containerName="istio-proxy"
Nov 02 21:50:25.136707 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.136687    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.183710 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.183630    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:50:25.188279 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.188252    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"]
Nov 02 21:50:25.189370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.189185    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:50:25.189370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.189204    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:50:25.189370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.189279    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:50:25.189370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.189319    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:50:25.270612 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.269653    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b970845-70c6-47e3-a1d6-f5d2527877f1-kube-api-access-mc25c\" (UniqueName: \"kubernetes.io/projected/9b970845-70c6-47e3-a1d6-f5d2527877f1-kube-api-access-mc25c\") pod \"9b970845-70c6-47e3-a1d6-f5d2527877f1\" (UID: \"9b970845-70c6-47e3-a1d6-f5d2527877f1\") "
Nov 02 21:50:25.270612 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.270357    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b970845-70c6-47e3-a1d6-f5d2527877f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b970845-70c6-47e3-a1d6-f5d2527877f1-plugins-config-volume\") pod \"9b970845-70c6-47e3-a1d6-f5d2527877f1\" (UID: \"9b970845-70c6-47e3-a1d6-f5d2527877f1\") "
Nov 02 21:50:25.271593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.271019    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b970845-70c6-47e3-a1d6-f5d2527877f1-plugins-config-volume" pod "9b970845-70c6-47e3-a1d6-f5d2527877f1" (UID: "9b970845-70c6-47e3-a1d6-f5d2527877f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:25.271593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.271234    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b21fdc-9dcsw\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.271593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.271317    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b21fdc-9dcsw\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.271593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.271417    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2cwv\" (UniqueName: \"kubernetes.io/projected/d5412d7f-dfae-4462-be30-b8e11967fb58-kube-api-access-k2cwv\") pod \"maas-api-revoke-sub-88b21fdc-9dcsw\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.271593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.271523    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b970845-70c6-47e3-a1d6-f5d2527877f1-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:25.275588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.275562    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b970845-70c6-47e3-a1d6-f5d2527877f1-kube-api-access-mc25c" pod "9b970845-70c6-47e3-a1d6-f5d2527877f1" (UID: "9b970845-70c6-47e3-a1d6-f5d2527877f1"). InnerVolumeSpecName "kube-api-access-mc25c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:25.295952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.295924    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"]
Nov 02 21:50:25.296442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.296408    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy" containerID="cri-o://c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e" gracePeriod=30
Nov 02 21:50:25.315443 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.315424    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:50:25.372321 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.372293    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2cwv\" (UniqueName: \"kubernetes.io/projected/d5412d7f-dfae-4462-be30-b8e11967fb58-kube-api-access-k2cwv\") pod \"maas-api-revoke-sub-88b21fdc-9dcsw\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.372438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.372419    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b21fdc-9dcsw\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.372509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.372486    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b21fdc-9dcsw\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.372652 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.372594    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc25c\" (UniqueName: \"kubernetes.io/projected/9b970845-70c6-47e3-a1d6-f5d2527877f1-kube-api-access-mc25c\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:25.373050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.373028    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b21fdc-9dcsw\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.374785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.374760    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b21fdc-9dcsw\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.463447 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.463355    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2cwv\" (UniqueName: \"kubernetes.io/projected/d5412d7f-dfae-4462-be30-b8e11967fb58-kube-api-access-k2cwv\") pod \"maas-api-revoke-sub-88b21fdc-9dcsw\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.473236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.473208    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6644f753-dd82-4c10-8775-72ce9e609997-kube-api-access-fq6dr\" (UniqueName: \"kubernetes.io/projected/6644f753-dd82-4c10-8775-72ce9e609997-kube-api-access-fq6dr\") pod \"6644f753-dd82-4c10-8775-72ce9e609997\" (UID: \"6644f753-dd82-4c10-8775-72ce9e609997\") "
Nov 02 21:50:25.473418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.473248    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6644f753-dd82-4c10-8775-72ce9e609997-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644f753-dd82-4c10-8775-72ce9e609997-plugins-config-volume\") pod \"6644f753-dd82-4c10-8775-72ce9e609997\" (UID: \"6644f753-dd82-4c10-8775-72ce9e609997\") "
Nov 02 21:50:25.473776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.473749    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6644f753-dd82-4c10-8775-72ce9e609997-plugins-config-volume" pod "6644f753-dd82-4c10-8775-72ce9e609997" (UID: "6644f753-dd82-4c10-8775-72ce9e609997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:25.475480 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.475451    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6644f753-dd82-4c10-8775-72ce9e609997-kube-api-access-fq6dr" pod "6644f753-dd82-4c10-8775-72ce9e609997" (UID: "6644f753-dd82-4c10-8775-72ce9e609997"). InnerVolumeSpecName "kube-api-access-fq6dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:25.575236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.575208    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6dr\" (UniqueName: \"kubernetes.io/projected/6644f753-dd82-4c10-8775-72ce9e609997-kube-api-access-fq6dr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:25.575236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.575233    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644f753-dd82-4c10-8775-72ce9e609997-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:25.753289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.753218    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:25.778815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.778786    4528 generic.go:356] "Generic (PLEG): container finished" podID="6644f753-dd82-4c10-8775-72ce9e609997" containerID="e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78" exitCode=0
Nov 02 21:50:25.778956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.778833    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"
Nov 02 21:50:25.778956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.778871    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k" event={"ID":"6644f753-dd82-4c10-8775-72ce9e609997","Type":"ContainerDied","Data":"e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78"}
Nov 02 21:50:25.778956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.778933    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k" event={"ID":"6644f753-dd82-4c10-8775-72ce9e609997","Type":"ContainerDied","Data":"5f385c153eadf7dd1c95ccc06e876b724d3032bbd06c498e61c90c289570959d"}
Nov 02 21:50:25.779104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.778967    4528 scope.go:122] "RemoveContainer" containerID="e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78"
Nov 02 21:50:25.780330 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.780306    4528 generic.go:356] "Generic (PLEG): container finished" podID="9b970845-70c6-47e3-a1d6-f5d2527877f1" containerID="e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226" exitCode=0
Nov 02 21:50:25.780444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.780356    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x" event={"ID":"9b970845-70c6-47e3-a1d6-f5d2527877f1","Type":"ContainerDied","Data":"e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226"}
Nov 02 21:50:25.780444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.780367    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"
Nov 02 21:50:25.780444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.780392    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x" event={"ID":"9b970845-70c6-47e3-a1d6-f5d2527877f1","Type":"ContainerDied","Data":"1755fec8c3cb4618b39362c510feb5799688fdb8377718eaf2fd74d39730f007"}
Nov 02 21:50:25.793329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.790687    4528 scope.go:122] "RemoveContainer" containerID="e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78"
Nov 02 21:50:25.793329 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:50:25.791019    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78\": container with ID starting with e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78 not found: ID does not exist" containerID="e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78"
Nov 02 21:50:25.793329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.791049    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78"} err="failed to get container status \"e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78\": rpc error: code = NotFound desc = could not find container \"e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78\": container with ID starting with e6c689fa87944c08f53f4d46f5a1a09b9320611d4e329419dcfc1974f5ca8b78 not found: ID does not exist"
Nov 02 21:50:25.793329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.791066    4528 scope.go:122] "RemoveContainer" containerID="e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226"
Nov 02 21:50:25.801950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.801933    4528 scope.go:122] "RemoveContainer" containerID="e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226"
Nov 02 21:50:25.802239 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:50:25.802217    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226\": container with ID starting with e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226 not found: ID does not exist" containerID="e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226"
Nov 02 21:50:25.802306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.802246    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226"} err="failed to get container status \"e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226\": rpc error: code = NotFound desc = could not find container \"e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226\": container with ID starting with e95aec90014c97adf7abb5089fff70a321db85a48ffe559b30b8057821b94226 not found: ID does not exist"
Nov 02 21:50:25.943565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.943537    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"]
Nov 02 21:50:25.946626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:25.946600    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8e90da-596c4b4d49-wd62x"]
Nov 02 21:50:26.066734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:26.066692    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"]
Nov 02 21:50:26.072322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:26.072298    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8e90da-7b7799f77f-l5c7k"]
Nov 02 21:50:26.128982 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:50:26.128945    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5412d7f_dfae_4462_be30_b8e11967fb58.slice/crio-760a0813996c0a33133bdf34a2c2e08618240db275a7df64ca0e5bfe6caad01a WatchSource:0}: Error finding container 760a0813996c0a33133bdf34a2c2e08618240db275a7df64ca0e5bfe6caad01a: Status 404 returned error can't find the container with id 760a0813996c0a33133bdf34a2c2e08618240db275a7df64ca0e5bfe6caad01a
Nov 02 21:50:26.131029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:26.131004    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"]
Nov 02 21:50:26.787669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:26.787614    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw" event={"ID":"d5412d7f-dfae-4462-be30-b8e11967fb58","Type":"ContainerStarted","Data":"f8ba52fd8121ecffa9b7f3acb29a77a4c495dd586745f6ef021d40c0260b9860"}
Nov 02 21:50:26.787857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:26.787682    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw" event={"ID":"d5412d7f-dfae-4462-be30-b8e11967fb58","Type":"ContainerStarted","Data":"760a0813996c0a33133bdf34a2c2e08618240db275a7df64ca0e5bfe6caad01a"}
Nov 02 21:50:26.901631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:26.901509    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw" podStartSLOduration=1.901494536 podStartE2EDuration="1.901494536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:26.900616518 +0000 UTC m=+2514.164152858" watchObservedRunningTime="2026-11-02 21:50:26.901494536 +0000 UTC m=+2514.165030890"
Nov 02 21:50:26.913608 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:26.913582    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6644f753-dd82-4c10-8775-72ce9e609997" path="/var/lib/kubelet/pods/6644f753-dd82-4c10-8775-72ce9e609997/volumes"
Nov 02 21:50:26.913974 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:26.913953    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b970845-70c6-47e3-a1d6-f5d2527877f1" path="/var/lib/kubelet/pods/9b970845-70c6-47e3-a1d6-f5d2527877f1/volumes"
Nov 02 21:50:27.796349 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:27.796314    4528 generic.go:356] "Generic (PLEG): container finished" podID="d5412d7f-dfae-4462-be30-b8e11967fb58" containerID="f8ba52fd8121ecffa9b7f3acb29a77a4c495dd586745f6ef021d40c0260b9860" exitCode=0
Nov 02 21:50:27.796759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:27.796353    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw" event={"ID":"d5412d7f-dfae-4462-be30-b8e11967fb58","Type":"ContainerDied","Data":"f8ba52fd8121ecffa9b7f3acb29a77a4c495dd586745f6ef021d40c0260b9860"}
Nov 02 21:50:28.954142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:28.954097    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:29.112870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.112784    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-openshift-ca\") pod \"d5412d7f-dfae-4462-be30-b8e11967fb58\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") "
Nov 02 21:50:29.112870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.112831    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-xks-ca\") pod \"d5412d7f-dfae-4462-be30-b8e11967fb58\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") "
Nov 02 21:50:29.113072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.112908    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5412d7f-dfae-4462-be30-b8e11967fb58-kube-api-access-k2cwv\" (UniqueName: \"kubernetes.io/projected/d5412d7f-dfae-4462-be30-b8e11967fb58-kube-api-access-k2cwv\") pod \"d5412d7f-dfae-4462-be30-b8e11967fb58\" (UID: \"d5412d7f-dfae-4462-be30-b8e11967fb58\") "
Nov 02 21:50:29.113178 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.113156    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-openshift-ca" pod "d5412d7f-dfae-4462-be30-b8e11967fb58" (UID: "d5412d7f-dfae-4462-be30-b8e11967fb58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:29.113413 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.113396    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:29.114833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.114808    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-xks-ca" pod "d5412d7f-dfae-4462-be30-b8e11967fb58" (UID: "d5412d7f-dfae-4462-be30-b8e11967fb58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:50:29.114922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.114855    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5412d7f-dfae-4462-be30-b8e11967fb58-kube-api-access-k2cwv" pod "d5412d7f-dfae-4462-be30-b8e11967fb58" (UID: "d5412d7f-dfae-4462-be30-b8e11967fb58"). InnerVolumeSpecName "kube-api-access-k2cwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:29.214842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.214808    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2cwv\" (UniqueName: \"kubernetes.io/projected/d5412d7f-dfae-4462-be30-b8e11967fb58-kube-api-access-k2cwv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:29.214842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.214836    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5412d7f-dfae-4462-be30-b8e11967fb58-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:29.808586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.808549    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw" event={"ID":"d5412d7f-dfae-4462-be30-b8e11967fb58","Type":"ContainerDied","Data":"760a0813996c0a33133bdf34a2c2e08618240db275a7df64ca0e5bfe6caad01a"}
Nov 02 21:50:29.808586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.808566    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"
Nov 02 21:50:29.808789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:29.808637    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760a0813996c0a33133bdf34a2c2e08618240db275a7df64ca0e5bfe6caad01a"
Nov 02 21:50:30.549824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.549799    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:50:30.730176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730084    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-certs\") pod \"f35d6134-28bb-4351-943d-0bd7d643dbe9\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") "
Nov 02 21:50:30.730176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730168    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-credential-socket\") pod \"f35d6134-28bb-4351-943d-0bd7d643dbe9\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") "
Nov 02 21:50:30.730420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730221    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-envoy\") pod \"f35d6134-28bb-4351-943d-0bd7d643dbe9\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") "
Nov 02 21:50:30.730420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730244    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-kube-api-access-fs6jg\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-kube-api-access-fs6jg\") pod \"f35d6134-28bb-4351-943d-0bd7d643dbe9\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") "
Nov 02 21:50:30.730420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730272    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35d6134-28bb-4351-943d-0bd7d643dbe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35d6134-28bb-4351-943d-0bd7d643dbe9-istiod-ca-cert\") pod \"f35d6134-28bb-4351-943d-0bd7d643dbe9\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") "
Nov 02 21:50:30.730420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730298    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-socket\") pod \"f35d6134-28bb-4351-943d-0bd7d643dbe9\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") "
Nov 02 21:50:30.730420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730350    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-certs" pod "f35d6134-28bb-4351-943d-0bd7d643dbe9" (UID: "f35d6134-28bb-4351-943d-0bd7d643dbe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:30.730420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730386    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-data\") pod \"f35d6134-28bb-4351-943d-0bd7d643dbe9\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") "
Nov 02 21:50:30.730764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730448    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-credential-socket" pod "f35d6134-28bb-4351-943d-0bd7d643dbe9" (UID: "f35d6134-28bb-4351-943d-0bd7d643dbe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:30.730764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730470    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-podinfo\") pod \"f35d6134-28bb-4351-943d-0bd7d643dbe9\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") "
Nov 02 21:50:30.730764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730500    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-token\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-token\") pod \"f35d6134-28bb-4351-943d-0bd7d643dbe9\" (UID: \"f35d6134-28bb-4351-943d-0bd7d643dbe9\") "
Nov 02 21:50:30.730764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730577    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-data" pod "f35d6134-28bb-4351-943d-0bd7d643dbe9" (UID: "f35d6134-28bb-4351-943d-0bd7d643dbe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:30.730764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730608    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-socket" pod "f35d6134-28bb-4351-943d-0bd7d643dbe9" (UID: "f35d6134-28bb-4351-943d-0bd7d643dbe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:30.730764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730687    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35d6134-28bb-4351-943d-0bd7d643dbe9-istiod-ca-cert" pod "f35d6134-28bb-4351-943d-0bd7d643dbe9" (UID: "f35d6134-28bb-4351-943d-0bd7d643dbe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:30.731086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730902    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:30.731086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730917    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35d6134-28bb-4351-943d-0bd7d643dbe9-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:30.731086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730932    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:30.731086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730954    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:30.731086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.730969    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:30.732666 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.732637    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-podinfo" pod "f35d6134-28bb-4351-943d-0bd7d643dbe9" (UID: "f35d6134-28bb-4351-943d-0bd7d643dbe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:50:30.732846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.732824    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-envoy" pod "f35d6134-28bb-4351-943d-0bd7d643dbe9" (UID: "f35d6134-28bb-4351-943d-0bd7d643dbe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:50:30.732914 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.732861    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-token" pod "f35d6134-28bb-4351-943d-0bd7d643dbe9" (UID: "f35d6134-28bb-4351-943d-0bd7d643dbe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:30.733065 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.733045    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-kube-api-access-fs6jg" pod "f35d6134-28bb-4351-943d-0bd7d643dbe9" (UID: "f35d6134-28bb-4351-943d-0bd7d643dbe9"). InnerVolumeSpecName "kube-api-access-fs6jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:30.815261 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.815230    4528 generic.go:356] "Generic (PLEG): container finished" podID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerID="c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e" exitCode=0
Nov 02 21:50:30.815402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.815287    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"
Nov 02 21:50:30.815402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.815321    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" event={"ID":"f35d6134-28bb-4351-943d-0bd7d643dbe9","Type":"ContainerDied","Data":"c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e"}
Nov 02 21:50:30.815402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.815382    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2" event={"ID":"f35d6134-28bb-4351-943d-0bd7d643dbe9","Type":"ContainerDied","Data":"af5aa59569792e4ed416a82f76c567d9dd9dffe5a9068673439b6da4423bc35a"}
Nov 02 21:50:30.815537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.815411    4528 scope.go:122] "RemoveContainer" containerID="c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e"
Nov 02 21:50:30.824973 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.824953    4528 scope.go:122] "RemoveContainer" containerID="bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb"
Nov 02 21:50:30.833459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.833435    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:30.833549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.833464    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs6jg\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-kube-api-access-fs6jg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:30.833549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.833478    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:30.833549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.833491    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35d6134-28bb-4351-943d-0bd7d643dbe9-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:30.834337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.834320    4528 scope.go:122] "RemoveContainer" containerID="c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e"
Nov 02 21:50:30.834617 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:50:30.834596    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e\": container with ID starting with c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e not found: ID does not exist" containerID="c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e"
Nov 02 21:50:30.834673 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.834626    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e"} err="failed to get container status \"c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e\": rpc error: code = NotFound desc = could not find container \"c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e\": container with ID starting with c84f8d97705c468d153cd735de5e40c5be57d9bb649acf57d3ac89730662cd6e not found: ID does not exist"
Nov 02 21:50:30.834673 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.834644    4528 scope.go:122] "RemoveContainer" containerID="bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb"
Nov 02 21:50:30.834859 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:50:30.834841    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb\": container with ID starting with bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb not found: ID does not exist" containerID="bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb"
Nov 02 21:50:30.834905 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.834865    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb"} err="failed to get container status \"bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb\": rpc error: code = NotFound desc = could not find container \"bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb\": container with ID starting with bebcde63cbc54fd0210683ac4f8a6c1e26829b68a9fc2210e54a441c22dabacb not found: ID does not exist"
Nov 02 21:50:30.983196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.982882    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"]
Nov 02 21:50:30.988673 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:30.988646    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cc0d6a-openshift-default-79b8d5b5d9-q8nm2"]
Nov 02 21:50:32.912612 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:32.912582    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" path="/var/lib/kubelet/pods/f35d6134-28bb-4351-943d-0bd7d643dbe9/volumes"
Nov 02 21:50:36.966565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.966530    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"]
Nov 02 21:50:36.967223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967204    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d5412d7f-dfae-4462-be30-b8e11967fb58" containerName="revoke-keys"
Nov 02 21:50:36.967336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967223    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5412d7f-dfae-4462-be30-b8e11967fb58" containerName="revoke-keys"
Nov 02 21:50:36.967336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967242    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9b970845-70c6-47e3-a1d6-f5d2527877f1" containerName="payload-processing"
Nov 02 21:50:36.967336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967250    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b970845-70c6-47e3-a1d6-f5d2527877f1" containerName="payload-processing"
Nov 02 21:50:36.967336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967264    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy"
Nov 02 21:50:36.967336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967271    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy"
Nov 02 21:50:36.967336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967288    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy"
Nov 02 21:50:36.967336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967294    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy"
Nov 02 21:50:36.967336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967309    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6644f753-dd82-4c10-8775-72ce9e609997" containerName="payload-pre-processing"
Nov 02 21:50:36.967336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967317    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6644f753-dd82-4c10-8775-72ce9e609997" containerName="payload-pre-processing"
Nov 02 21:50:36.967786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967431    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5412d7f-dfae-4462-be30-b8e11967fb58" containerName="revoke-keys"
Nov 02 21:50:36.967786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967446    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b970845-70c6-47e3-a1d6-f5d2527877f1" containerName="payload-processing"
Nov 02 21:50:36.967786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967454    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy"
Nov 02 21:50:36.967786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967464    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35d6134-28bb-4351-943d-0bd7d643dbe9" containerName="istio-proxy"
Nov 02 21:50:36.967786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.967476    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6644f753-dd82-4c10-8775-72ce9e609997" containerName="payload-pre-processing"
Nov 02 21:50:36.972491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.972472    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:36.997020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.996998    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bea45e1b-66t4b\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:36.997131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.997073    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldk8s\" (UniqueName: \"kubernetes.io/projected/123c5c7b-2fc5-4875-b20f-f532461bd641-kube-api-access-ldk8s\") pod \"maas-api-revoke-sub-bea45e1b-66t4b\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:36.997223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:36.997166    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bea45e1b-66t4b\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:37.022215 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.022190    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"]
Nov 02 21:50:37.069790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.069760    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:50:37.076192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.076168    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:50:37.076307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.076168    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:50:37.076307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.076178    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:50:37.097690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.097669    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldk8s\" (UniqueName: \"kubernetes.io/projected/123c5c7b-2fc5-4875-b20f-f532461bd641-kube-api-access-ldk8s\") pod \"maas-api-revoke-sub-bea45e1b-66t4b\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:37.097794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.097719    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bea45e1b-66t4b\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:37.097794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.097750    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bea45e1b-66t4b\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:37.098374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.098352    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bea45e1b-66t4b\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:37.100060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.100039    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bea45e1b-66t4b\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:37.154683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.154656    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldk8s\" (UniqueName: \"kubernetes.io/projected/123c5c7b-2fc5-4875-b20f-f532461bd641-kube-api-access-ldk8s\") pod \"maas-api-revoke-sub-bea45e1b-66t4b\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:37.285520 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.285495    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:37.453479 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.453446    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"]
Nov 02 21:50:37.458410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.458391    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.501206 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.501176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8babdc-4005-4220-9023-a91d3b34fd2e-tls-certs\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.501363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.501285    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-dshm\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.501363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.501334    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-home\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.501458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.501405    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-tmp-dir\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.501458 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.501440    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gdk\" (UniqueName: \"kubernetes.io/projected/ff8babdc-4005-4220-9023-a91d3b34fd2e-kube-api-access-f8gdk\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.501544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.501474    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-model-cache\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.510215 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.510183    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"]
Nov 02 21:50:37.510834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.510807    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-aa8aff\"/\"kube-root-ca.crt\""
Nov 02 21:50:37.510946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.510807    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-aa8aff\"/\"openshift-service-ca.crt\""
Nov 02 21:50:37.510946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.510929    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-aa8aff\"/\"praxis-route-aa8aff-kserve-self-signed-certs\""
Nov 02 21:50:37.511066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.510982    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-aa8aff\"/\"default-dockercfg-mwf5l\""
Nov 02 21:50:37.617179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.608032    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-home\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.617179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.608088    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-tmp-dir\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.617179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.608138    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gdk\" (UniqueName: \"kubernetes.io/projected/ff8babdc-4005-4220-9023-a91d3b34fd2e-kube-api-access-f8gdk\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.617179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.608184    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-model-cache\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.617179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.608281    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8babdc-4005-4220-9023-a91d3b34fd2e-tls-certs\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.617179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.608363    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-dshm\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.617179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.612268    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-tmp-dir\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.617179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.612804    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-home\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.617179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.613044    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-model-cache\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.620888 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.619836    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8babdc-4005-4220-9023-a91d3b34fd2e-tls-certs\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.621312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.621285    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-dshm\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.670182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.670156    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gdk\" (UniqueName: \"kubernetes.io/projected/ff8babdc-4005-4220-9023-a91d3b34fd2e-kube-api-access-f8gdk\") pod \"praxis-route-aa8aff-kserve-868649bc55-92kmq\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") " pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.680176 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:50:37.680153    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123c5c7b_2fc5_4875_b20f_f532461bd641.slice/crio-564c26d068f798e84b77d08a8c641e1d3dfd2283057413de4b20ba40d82bb3a2 WatchSource:0}: Error finding container 564c26d068f798e84b77d08a8c641e1d3dfd2283057413de4b20ba40d82bb3a2: Status 404 returned error can't find the container with id 564c26d068f798e84b77d08a8c641e1d3dfd2283057413de4b20ba40d82bb3a2
Nov 02 21:50:37.682604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.682580    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"]
Nov 02 21:50:37.770491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.770464    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:37.855582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:37.855540    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b" event={"ID":"123c5c7b-2fc5-4875-b20f-f532461bd641","Type":"ContainerStarted","Data":"564c26d068f798e84b77d08a8c641e1d3dfd2283057413de4b20ba40d82bb3a2"}
Nov 02 21:50:38.142696 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:50:38.142662    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff8babdc_4005_4220_9023_a91d3b34fd2e.slice/crio-7fd1c6a26c1d830238929da2f0315d59980f7481fbc5b8b89f764881ec6f4f74 WatchSource:0}: Error finding container 7fd1c6a26c1d830238929da2f0315d59980f7481fbc5b8b89f764881ec6f4f74: Status 404 returned error can't find the container with id 7fd1c6a26c1d830238929da2f0315d59980f7481fbc5b8b89f764881ec6f4f74
Nov 02 21:50:38.146570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:38.146539    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"]
Nov 02 21:50:38.862622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:38.862529    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq" event={"ID":"ff8babdc-4005-4220-9023-a91d3b34fd2e","Type":"ContainerStarted","Data":"070881ad63ba9b36dd0e71dd7c1805f489250b1ffd0feec0229b2532b4453d38"}
Nov 02 21:50:38.862622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:38.862574    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq" event={"ID":"ff8babdc-4005-4220-9023-a91d3b34fd2e","Type":"ContainerStarted","Data":"7fd1c6a26c1d830238929da2f0315d59980f7481fbc5b8b89f764881ec6f4f74"}
Nov 02 21:50:38.864157 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:38.864123    4528 generic.go:356] "Generic (PLEG): container finished" podID="123c5c7b-2fc5-4875-b20f-f532461bd641" containerID="42035cfb090eb971e62e36d0847a23a93905c9d28a4c3c17e7122be2fb72619f" exitCode=0
Nov 02 21:50:38.864289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:38.864166    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b" event={"ID":"123c5c7b-2fc5-4875-b20f-f532461bd641","Type":"ContainerDied","Data":"42035cfb090eb971e62e36d0847a23a93905c9d28a4c3c17e7122be2fb72619f"}
Nov 02 21:50:38.978683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:38.978634    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq" podStartSLOduration=1.978618907 podStartE2EDuration="1.978618907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:38.976246972 +0000 UTC m=+2526.239783288" watchObservedRunningTime="2026-11-02 21:50:38.978618907 +0000 UTC m=+2526.242155223"
Nov 02 21:50:39.771256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:39.771223    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:40.007792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.007770    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:40.033304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.033247    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123c5c7b-2fc5-4875-b20f-f532461bd641-kube-api-access-ldk8s\" (UniqueName: \"kubernetes.io/projected/123c5c7b-2fc5-4875-b20f-f532461bd641-kube-api-access-ldk8s\") pod \"123c5c7b-2fc5-4875-b20f-f532461bd641\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") "
Nov 02 21:50:40.033408 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.033303    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-openshift-ca\") pod \"123c5c7b-2fc5-4875-b20f-f532461bd641\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") "
Nov 02 21:50:40.033482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.033467    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-xks-ca\") pod \"123c5c7b-2fc5-4875-b20f-f532461bd641\" (UID: \"123c5c7b-2fc5-4875-b20f-f532461bd641\") "
Nov 02 21:50:40.033688 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.033666    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-openshift-ca" pod "123c5c7b-2fc5-4875-b20f-f532461bd641" (UID: "123c5c7b-2fc5-4875-b20f-f532461bd641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:40.033857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.033844    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:40.035742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.035719    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-xks-ca" pod "123c5c7b-2fc5-4875-b20f-f532461bd641" (UID: "123c5c7b-2fc5-4875-b20f-f532461bd641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:50:40.036171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.036153    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123c5c7b-2fc5-4875-b20f-f532461bd641-kube-api-access-ldk8s" pod "123c5c7b-2fc5-4875-b20f-f532461bd641" (UID: "123c5c7b-2fc5-4875-b20f-f532461bd641"). InnerVolumeSpecName "kube-api-access-ldk8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:40.135278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.135251    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123c5c7b-2fc5-4875-b20f-f532461bd641-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:40.135278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.135280    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldk8s\" (UniqueName: \"kubernetes.io/projected/123c5c7b-2fc5-4875-b20f-f532461bd641-kube-api-access-ldk8s\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:40.877409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.877380    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"
Nov 02 21:50:40.877812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.877414    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b" event={"ID":"123c5c7b-2fc5-4875-b20f-f532461bd641","Type":"ContainerDied","Data":"564c26d068f798e84b77d08a8c641e1d3dfd2283057413de4b20ba40d82bb3a2"}
Nov 02 21:50:40.877812 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.877460    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="564c26d068f798e84b77d08a8c641e1d3dfd2283057413de4b20ba40d82bb3a2"
Nov 02 21:50:40.984681 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.984638    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"]
Nov 02 21:50:40.985427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.985403    4528 cpu_manager.go:414] "containerMap: removing container" podUID="123c5c7b-2fc5-4875-b20f-f532461bd641" containerName="revoke-keys"
Nov 02 21:50:40.985427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.985427    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123c5c7b-2fc5-4875-b20f-f532461bd641" containerName="revoke-keys"
Nov 02 21:50:40.985616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.985604    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="123c5c7b-2fc5-4875-b20f-f532461bd641" containerName="revoke-keys"
Nov 02 21:50:40.989511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:40.989488    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.041310 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.041276    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"]
Nov 02 21:50:41.043361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.043336    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:50:41.043525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.043439    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:50:41.043525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.043478    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:50:41.043525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.043515    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:50:41.046778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.046751    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24cc\" (UniqueName: \"kubernetes.io/projected/ecc6c624-333d-4f0d-8938-93c3116e723d-kube-api-access-f24cc\") pod \"maas-api-revoke-sub-322ba04d-phv7l\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.046937 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.046917    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322ba04d-phv7l\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.047062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.046963    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322ba04d-phv7l\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.148659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.148569    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322ba04d-phv7l\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.148659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.148636    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322ba04d-phv7l\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.148903 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.148730    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24cc\" (UniqueName: \"kubernetes.io/projected/ecc6c624-333d-4f0d-8938-93c3116e723d-kube-api-access-f24cc\") pod \"maas-api-revoke-sub-322ba04d-phv7l\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.149477 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.149445    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322ba04d-phv7l\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.151163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.151137    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322ba04d-phv7l\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.205663 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.205633    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24cc\" (UniqueName: \"kubernetes.io/projected/ecc6c624-333d-4f0d-8938-93c3116e723d-kube-api-access-f24cc\") pod \"maas-api-revoke-sub-322ba04d-phv7l\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.306671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.306642    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:41.684893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.684865    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"]
Nov 02 21:50:41.883745 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:41.883703    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l" event={"ID":"ecc6c624-333d-4f0d-8938-93c3116e723d","Type":"ContainerStarted","Data":"f13b57dc2bcbdb622cf0641fdf6436785e2741c05ef6303a1dcf69b33d05ea19"}
Nov 02 21:50:42.890771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:42.890685    4528 generic.go:356] "Generic (PLEG): container finished" podID="ecc6c624-333d-4f0d-8938-93c3116e723d" containerID="480f4337282cc20603b65b407e61a88bfc96e41825a45fa1d170149c63f6127b" exitCode=0
Nov 02 21:50:42.891133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:42.890769    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l" event={"ID":"ecc6c624-333d-4f0d-8938-93c3116e723d","Type":"ContainerDied","Data":"480f4337282cc20603b65b407e61a88bfc96e41825a45fa1d170149c63f6127b"}
Nov 02 21:50:44.061003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.060982    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:44.079155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.079133    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc6c624-333d-4f0d-8938-93c3116e723d-kube-api-access-f24cc\" (UniqueName: \"kubernetes.io/projected/ecc6c624-333d-4f0d-8938-93c3116e723d-kube-api-access-f24cc\") pod \"ecc6c624-333d-4f0d-8938-93c3116e723d\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") "
Nov 02 21:50:44.079281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.079199    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-openshift-ca\") pod \"ecc6c624-333d-4f0d-8938-93c3116e723d\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") "
Nov 02 21:50:44.079348 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.079336    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-xks-ca\") pod \"ecc6c624-333d-4f0d-8938-93c3116e723d\" (UID: \"ecc6c624-333d-4f0d-8938-93c3116e723d\") "
Nov 02 21:50:44.079554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.079531    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-openshift-ca" pod "ecc6c624-333d-4f0d-8938-93c3116e723d" (UID: "ecc6c624-333d-4f0d-8938-93c3116e723d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:44.081066 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.081045    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc6c624-333d-4f0d-8938-93c3116e723d-kube-api-access-f24cc" pod "ecc6c624-333d-4f0d-8938-93c3116e723d" (UID: "ecc6c624-333d-4f0d-8938-93c3116e723d"). InnerVolumeSpecName "kube-api-access-f24cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:44.081168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.081119    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-xks-ca" pod "ecc6c624-333d-4f0d-8938-93c3116e723d" (UID: "ecc6c624-333d-4f0d-8938-93c3116e723d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:50:44.111891 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:50:44.111858    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:50:44.181188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.181121    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:44.181188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.181155    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f24cc\" (UniqueName: \"kubernetes.io/projected/ecc6c624-333d-4f0d-8938-93c3116e723d-kube-api-access-f24cc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:44.181188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.181170    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc6c624-333d-4f0d-8938-93c3116e723d-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:44.903040 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.903010    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"
Nov 02 21:50:44.903040 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.903027    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l" event={"ID":"ecc6c624-333d-4f0d-8938-93c3116e723d","Type":"ContainerDied","Data":"f13b57dc2bcbdb622cf0641fdf6436785e2741c05ef6303a1dcf69b33d05ea19"}
Nov 02 21:50:44.903320 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:44.903073    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13b57dc2bcbdb622cf0641fdf6436785e2741c05ef6303a1dcf69b33d05ea19"
Nov 02 21:50:46.120541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.120505    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"]
Nov 02 21:50:46.121027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.121012    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ecc6c624-333d-4f0d-8938-93c3116e723d" containerName="revoke-keys"
Nov 02 21:50:46.121027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.121027    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc6c624-333d-4f0d-8938-93c3116e723d" containerName="revoke-keys"
Nov 02 21:50:46.121165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.121154    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc6c624-333d-4f0d-8938-93c3116e723d" containerName="revoke-keys"
Nov 02 21:50:46.128674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.128632    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.170406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.170379    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"]
Nov 02 21:50:46.174104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.174088    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.182323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.182306    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f51151-openshift-default-dockercfg-5nn46\""
Nov 02 21:50:46.200738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.200710    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0524738-f680-4ece-a83d-61b302af5aff-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.200851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.200764    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.200851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.200824    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.200945 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.200913    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97872b-bd19-40fe-9f23-cf44c308603c-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.200995 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.200951    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.201035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201006    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.201071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201047    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.201071 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201065    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.201192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201126    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.201252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201212    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.201309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201250    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.201365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201324    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.201365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201352    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0524738-f680-4ece-a83d-61b302af5aff-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.201452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201381    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7hf\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-kube-api-access-pj7hf\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.201452 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201439    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.201514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201487    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znt7g\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-kube-api-access-znt7g\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.201514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201507    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.201573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.201538    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.221996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.221970    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"]
Nov 02 21:50:46.234676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.234651    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"]
Nov 02 21:50:46.302762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.302733    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.302927 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.302774    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.302927 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.302814    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.302927 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.302838    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0524738-f680-4ece-a83d-61b302af5aff-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.302927 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.302866    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7hf\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-kube-api-access-pj7hf\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.303213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303118    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303152    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303191    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znt7g\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-kube-api-access-znt7g\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303213 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303210    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.303439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303260    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.303439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303309    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0524738-f680-4ece-a83d-61b302af5aff-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303317    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303374    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303405    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303452    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303454    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97872b-bd19-40fe-9f23-cf44c308603c-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.303690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303496    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.303690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303531    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.303690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303559    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0524738-f680-4ece-a83d-61b302af5aff-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303581    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.303690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303592    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.303690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303609    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.303690 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303670    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.304163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303813    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.304163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.303966    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.304163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.304047    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.304163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.304083    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97872b-bd19-40fe-9f23-cf44c308603c-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.304322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.304198    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.305684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.305655    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.305784 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.305726    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.306003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.305986    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.306041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.306023    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0524738-f680-4ece-a83d-61b302af5aff-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.363601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.363574    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znt7g\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-kube-api-access-znt7g\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.363716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.363574    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7hf\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-kube-api-access-pj7hf\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.363909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.363885    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.364581 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.364565    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.442371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.442304    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:46.486698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.486664    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:46.862505 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:50:46.862467    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0524738_f680_4ece_a83d_61b302af5aff.slice/crio-ffdafeba81ab572d5817cddd09abf6321007f371ed33908468370888dad378ad WatchSource:0}: Error finding container ffdafeba81ab572d5817cddd09abf6321007f371ed33908468370888dad378ad: Status 404 returned error can't find the container with id ffdafeba81ab572d5817cddd09abf6321007f371ed33908468370888dad378ad
Nov 02 21:50:46.864787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.864766    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"]
Nov 02 21:50:46.864874 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.864826    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:50:46.864909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.864881    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:50:46.864950 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.864910    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:50:46.915956 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.915927    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" event={"ID":"c0524738-f680-4ece-a83d-61b302af5aff","Type":"ContainerStarted","Data":"ffdafeba81ab572d5817cddd09abf6321007f371ed33908468370888dad378ad"}
Nov 02 21:50:46.916571 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.916555    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"]
Nov 02 21:50:46.926163 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:50:46.926141    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff97872b_bd19_40fe_9f23_cf44c308603c.slice/crio-09e481108d97f21f13782b0ffb4952918dd938c4b20bedab0def47a3dd14df6e WatchSource:0}: Error finding container 09e481108d97f21f13782b0ffb4952918dd938c4b20bedab0def47a3dd14df6e: Status 404 returned error can't find the container with id 09e481108d97f21f13782b0ffb4952918dd938c4b20bedab0def47a3dd14df6e
Nov 02 21:50:46.928321 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.928293    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:50:46.928429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.928365    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:50:46.928429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:46.928409    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:50:47.771554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:47.771513    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:47.787823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:47.787786    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:47.952177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:47.951525    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" event={"ID":"ff97872b-bd19-40fe-9f23-cf44c308603c","Type":"ContainerStarted","Data":"f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5"}
Nov 02 21:50:47.952177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:47.951588    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" event={"ID":"ff97872b-bd19-40fe-9f23-cf44c308603c","Type":"ContainerStarted","Data":"09e481108d97f21f13782b0ffb4952918dd938c4b20bedab0def47a3dd14df6e"}
Nov 02 21:50:47.955295 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:47.955254    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" event={"ID":"c0524738-f680-4ece-a83d-61b302af5aff","Type":"ContainerStarted","Data":"a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d"}
Nov 02 21:50:47.966624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:47.966599    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:50:48.072306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:48.072208    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podStartSLOduration=2.072192579 podStartE2EDuration="2.072192579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:48.069325661 +0000 UTC m=+2535.332861978" watchObservedRunningTime="2026-11-02 21:50:48.072192579 +0000 UTC m=+2535.335728894"
Nov 02 21:50:48.195233 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:48.195178    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" podStartSLOduration=2.195159233 podStartE2EDuration="2.195159233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:48.191184687 +0000 UTC m=+2535.454721013" watchObservedRunningTime="2026-11-02 21:50:48.195159233 +0000 UTC m=+2535.458695550"
Nov 02 21:50:48.453192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:48.452514    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:48.464133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:48.458029    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 02 21:50:48.464133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:48.458088    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 02 21:50:48.488000 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:48.487684    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:48.489136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:48.489099    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 21:50:48.489258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:48.489155    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 21:50:49.444132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:49.443498    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 02 21:50:49.444132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:49.443564    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 02 21:50:49.487188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:49.487151    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 21:50:49.487359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:49.487220    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 21:50:50.445132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:50.444517    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 02 21:50:50.445132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:50.444597    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 02 21:50:50.487985 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:50.487958    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 21:50:50.488219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:50.488197    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 21:50:51.445549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.445516    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 02 21:50:51.446153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.446100    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 02 21:50:51.487776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.487744    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 21:50:51.487942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.487835    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 21:50:51.583688 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.583650    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:50:51.769619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.769583    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"]
Nov 02 21:50:51.773795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.773778    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:51.824270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.823958    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b962dec-f8lxt\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:51.824270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.824045    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b962dec-f8lxt\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:51.824270 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.824077    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkv7p\" (UniqueName: \"kubernetes.io/projected/acd349a2-13f8-40a2-8d0f-996ba522b27f-kube-api-access-kkv7p\") pod \"maas-api-revoke-sub-1b962dec-f8lxt\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:51.837344 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.837317    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"]
Nov 02 21:50:51.844132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.839408    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:50:51.844132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.839563    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:50:51.844132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.840214    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:50:51.844132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.840311    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:50:51.940133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.938829    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b962dec-f8lxt\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:51.940133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.938908    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b962dec-f8lxt\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:51.940133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.938938    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkv7p\" (UniqueName: \"kubernetes.io/projected/acd349a2-13f8-40a2-8d0f-996ba522b27f-kube-api-access-kkv7p\") pod \"maas-api-revoke-sub-1b962dec-f8lxt\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:51.955272 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.955240    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b962dec-f8lxt\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:51.957520 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:51.957495    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b962dec-f8lxt\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:52.025179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:52.021210    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkv7p\" (UniqueName: \"kubernetes.io/projected/acd349a2-13f8-40a2-8d0f-996ba522b27f-kube-api-access-kkv7p\") pod \"maas-api-revoke-sub-1b962dec-f8lxt\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:52.088945 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:52.088300    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:52.466001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:52.465554    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 02 21:50:52.466001 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:52.465618    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 02 21:50:52.487665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:52.487629    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 21:50:52.488093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:52.488019    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 21:50:52.655931 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:52.655843    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"]
Nov 02 21:50:53.003189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.002554    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt" event={"ID":"acd349a2-13f8-40a2-8d0f-996ba522b27f","Type":"ContainerStarted","Data":"564f0ee923268c0f470e265e4f22bf8814f100b4e8d38c17bb75bc577ab44cb5"}
Nov 02 21:50:53.141181 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.141138    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"]
Nov 02 21:50:53.147647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.147620    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.186977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.186951    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"]
Nov 02 21:50:53.190511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.190476    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-536b0d42-76plt\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.190891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.190862    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-536b0d42-76plt\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.191162 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.191141    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nvgm\" (UniqueName: \"kubernetes.io/projected/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-kube-api-access-2nvgm\") pod \"maas-api-revoke-sub-536b0d42-76plt\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.293455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.293420    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-536b0d42-76plt\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.294143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.293726    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nvgm\" (UniqueName: \"kubernetes.io/projected/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-kube-api-access-2nvgm\") pod \"maas-api-revoke-sub-536b0d42-76plt\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.294143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.293857    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-536b0d42-76plt\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.294728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.294701    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-536b0d42-76plt\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.295933 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.295915    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-536b0d42-76plt\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.353142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.352493    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nvgm\" (UniqueName: \"kubernetes.io/projected/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-kube-api-access-2nvgm\") pod \"maas-api-revoke-sub-536b0d42-76plt\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.443277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.443232    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 02 21:50:53.443455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.443297    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 02 21:50:53.471101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.471068    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:53.488746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.488459    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 21:50:53.488746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.488516    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 21:50:53.940811 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:53.939967    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"]
Nov 02 21:50:54.026132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:54.025539    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt" event={"ID":"e87b2f26-35bb-41d9-8f9b-6b4d71a22624","Type":"ContainerStarted","Data":"b92f66380cab1401cb5d0ec7a2ecfc0aaa6afde2053c8f825f7dea4f633d4878"}
Nov 02 21:50:54.028316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:54.027833    4528 generic.go:356] "Generic (PLEG): container finished" podID="acd349a2-13f8-40a2-8d0f-996ba522b27f" containerID="60b96bdde435107b85a574d6f4af8396ec66d37330abbc3cbbac292cb964ae25" exitCode=0
Nov 02 21:50:54.028316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:54.027918    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt" event={"ID":"acd349a2-13f8-40a2-8d0f-996ba522b27f","Type":"ContainerDied","Data":"60b96bdde435107b85a574d6f4af8396ec66d37330abbc3cbbac292cb964ae25"}
Nov 02 21:50:55.050145 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.049813    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt" event={"ID":"e87b2f26-35bb-41d9-8f9b-6b4d71a22624","Type":"ContainerStarted","Data":"6dd0f2d5e156c99b2278b0ad071133aa67acb96c83d71b975c57608ebd40eadd"}
Nov 02 21:50:55.168243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.168211    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:55.168648 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.168625    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:55.169885 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.169868    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 21:50:55.201336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.200545    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt" podStartSLOduration=2.20052388 podStartE2EDuration="2.20052388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:55.178871862 +0000 UTC m=+2542.442408180" watchObservedRunningTime="2026-11-02 21:50:55.20052388 +0000 UTC m=+2542.464060197"
Nov 02 21:50:55.361259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.360086    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:55.461844 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.461793    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-xks-ca\") pod \"acd349a2-13f8-40a2-8d0f-996ba522b27f\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") "
Nov 02 21:50:55.462059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.461968    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-openshift-ca\") pod \"acd349a2-13f8-40a2-8d0f-996ba522b27f\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") "
Nov 02 21:50:55.462059 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.462024    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd349a2-13f8-40a2-8d0f-996ba522b27f-kube-api-access-kkv7p\" (UniqueName: \"kubernetes.io/projected/acd349a2-13f8-40a2-8d0f-996ba522b27f-kube-api-access-kkv7p\") pod \"acd349a2-13f8-40a2-8d0f-996ba522b27f\" (UID: \"acd349a2-13f8-40a2-8d0f-996ba522b27f\") "
Nov 02 21:50:55.463195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.463164    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-openshift-ca" pod "acd349a2-13f8-40a2-8d0f-996ba522b27f" (UID: "acd349a2-13f8-40a2-8d0f-996ba522b27f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:55.465200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.465171    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd349a2-13f8-40a2-8d0f-996ba522b27f-kube-api-access-kkv7p" pod "acd349a2-13f8-40a2-8d0f-996ba522b27f" (UID: "acd349a2-13f8-40a2-8d0f-996ba522b27f"). InnerVolumeSpecName "kube-api-access-kkv7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:55.465513 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.465488    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-xks-ca" pod "acd349a2-13f8-40a2-8d0f-996ba522b27f" (UID: "acd349a2-13f8-40a2-8d0f-996ba522b27f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:50:55.489405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.488601    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:50:55.489405 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.488661    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:50:55.563226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.563174    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:55.563226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.563214    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkv7p\" (UniqueName: \"kubernetes.io/projected/acd349a2-13f8-40a2-8d0f-996ba522b27f-kube-api-access-kkv7p\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:55.563226 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:55.563225    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd349a2-13f8-40a2-8d0f-996ba522b27f-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:56.071131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.071065    4528 generic.go:356] "Generic (PLEG): container finished" podID="e87b2f26-35bb-41d9-8f9b-6b4d71a22624" containerID="6dd0f2d5e156c99b2278b0ad071133aa67acb96c83d71b975c57608ebd40eadd" exitCode=0
Nov 02 21:50:56.071755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.071197    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt" event={"ID":"e87b2f26-35bb-41d9-8f9b-6b4d71a22624","Type":"ContainerDied","Data":"6dd0f2d5e156c99b2278b0ad071133aa67acb96c83d71b975c57608ebd40eadd"}
Nov 02 21:50:56.075090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.074059    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"
Nov 02 21:50:56.075247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.075170    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt" event={"ID":"acd349a2-13f8-40a2-8d0f-996ba522b27f","Type":"ContainerDied","Data":"564f0ee923268c0f470e265e4f22bf8814f100b4e8d38c17bb75bc577ab44cb5"}
Nov 02 21:50:56.075247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.075213    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="564f0ee923268c0f470e265e4f22bf8814f100b4e8d38c17bb75bc577ab44cb5"
Nov 02 21:50:56.533132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.531221    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:50:56.533132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.531287    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:50:56.898717 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.898633    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:56.899043 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.899023    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:56.899786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:56.899770    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 21:50:57.292415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.292389    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:57.387244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.387210    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-xks-ca\") pod \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") "
Nov 02 21:50:57.387499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.387351    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-kube-api-access-2nvgm\" (UniqueName: \"kubernetes.io/projected/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-kube-api-access-2nvgm\") pod \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") "
Nov 02 21:50:57.387499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.387485    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-openshift-ca\") pod \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\" (UID: \"e87b2f26-35bb-41d9-8f9b-6b4d71a22624\") "
Nov 02 21:50:57.387852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.387828    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-openshift-ca" pod "e87b2f26-35bb-41d9-8f9b-6b4d71a22624" (UID: "e87b2f26-35bb-41d9-8f9b-6b4d71a22624"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:50:57.389459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.389429    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-xks-ca" pod "e87b2f26-35bb-41d9-8f9b-6b4d71a22624" (UID: "e87b2f26-35bb-41d9-8f9b-6b4d71a22624"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:50:57.389882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.389861    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-kube-api-access-2nvgm" pod "e87b2f26-35bb-41d9-8f9b-6b4d71a22624" (UID: "e87b2f26-35bb-41d9-8f9b-6b4d71a22624"). InnerVolumeSpecName "kube-api-access-2nvgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:50:57.489570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.489490    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:57.489570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.489521    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:57.489570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:57.489531    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nvgm\" (UniqueName: \"kubernetes.io/projected/e87b2f26-35bb-41d9-8f9b-6b4d71a22624-kube-api-access-2nvgm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:50:58.085542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:58.085502    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt" event={"ID":"e87b2f26-35bb-41d9-8f9b-6b4d71a22624","Type":"ContainerDied","Data":"b92f66380cab1401cb5d0ec7a2ecfc0aaa6afde2053c8f825f7dea4f633d4878"}
Nov 02 21:50:58.085542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:58.085541    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"
Nov 02 21:50:58.085763 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:50:58.085554    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b92f66380cab1401cb5d0ec7a2ecfc0aaa6afde2053c8f825f7dea4f633d4878"
Nov 02 21:51:02.109800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:02.109732    4528 generic.go:356] "Generic (PLEG): container finished" podID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerID="78323832e5ffd012c315bf361e38803f14d376ab18f74471d764ca2407edef06" exitCode=0
Nov 02 21:51:02.110211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:02.109800    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" event={"ID":"8cbf6236-95c3-4847-9650-f5de5af07ad2","Type":"ContainerDied","Data":"78323832e5ffd012c315bf361e38803f14d376ab18f74471d764ca2407edef06"}
Nov 02 21:51:02.110211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:02.110201    4528 scope.go:122] "RemoveContainer" containerID="78323832e5ffd012c315bf361e38803f14d376ab18f74471d764ca2407edef06"
Nov 02 21:51:02.111668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:02.111637    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:02.111762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:02.111707    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:02.111762 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:02.111744    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:03.133036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:03.133001    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" event={"ID":"8cbf6236-95c3-4847-9650-f5de5af07ad2","Type":"ContainerStarted","Data":"25171310f502d4f2cf575d94f65652e88b6c7233930515e9dea49c02ebebb601"}
Nov 02 21:51:03.759755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:03.759720    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:51:03.760970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:03.760946    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 02 21:51:03.761101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:03.761013    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 02 21:51:04.759375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:04.759341    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 02 21:51:04.759789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:04.759420    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 02 21:51:05.759400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:05.759365    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 02 21:51:05.759789 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:05.759420    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 02 21:51:06.352043 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:06.352013    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:51:06.759825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:06.759744    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 02 21:51:06.759825 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:06.759796    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 02 21:51:07.760219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:07.760183    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 02 21:51:07.760630 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:07.760237    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 02 21:51:08.511576 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:08.511545    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"]
Nov 02 21:51:08.511833 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:08.511791    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy" containerID="cri-o://6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88" gracePeriod=30
Nov 02 21:51:08.764132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:08.764037    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:51:08.765076 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:08.765054    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:51:13.116324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.116292    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"]
Nov 02 21:51:13.116754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.116741    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e87b2f26-35bb-41d9-8f9b-6b4d71a22624" containerName="revoke-keys"
Nov 02 21:51:13.116802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.116755    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87b2f26-35bb-41d9-8f9b-6b4d71a22624" containerName="revoke-keys"
Nov 02 21:51:13.116802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.116769    4528 cpu_manager.go:414] "containerMap: removing container" podUID="acd349a2-13f8-40a2-8d0f-996ba522b27f" containerName="revoke-keys"
Nov 02 21:51:13.116802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.116774    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd349a2-13f8-40a2-8d0f-996ba522b27f" containerName="revoke-keys"
Nov 02 21:51:13.116901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.116865    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87b2f26-35bb-41d9-8f9b-6b4d71a22624" containerName="revoke-keys"
Nov 02 21:51:13.116901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.116872    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd349a2-13f8-40a2-8d0f-996ba522b27f" containerName="revoke-keys"
Nov 02 21:51:13.120423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.120408    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.174465 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.174435    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"]
Nov 02 21:51:13.209715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.209687    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e882f3d1-bxwzr\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.209860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.209752    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e882f3d1-bxwzr\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.209860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.209832    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78rsk\" (UniqueName: \"kubernetes.io/projected/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-kube-api-access-78rsk\") pod \"maas-api-revoke-sub-e882f3d1-bxwzr\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.221069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.221046    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:51:13.221175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.221091    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:51:13.222708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.222694    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:51:13.222777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.222760    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:51:13.310657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.310625    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e882f3d1-bxwzr\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.310828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.310688    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e882f3d1-bxwzr\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.310828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.310730    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78rsk\" (UniqueName: \"kubernetes.io/projected/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-kube-api-access-78rsk\") pod \"maas-api-revoke-sub-e882f3d1-bxwzr\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.311293 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.311272    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e882f3d1-bxwzr\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.313022 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.313002    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e882f3d1-bxwzr\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.367933 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.367873    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78rsk\" (UniqueName: \"kubernetes.io/projected/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-kube-api-access-78rsk\") pod \"maas-api-revoke-sub-e882f3d1-bxwzr\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.433089 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.433066    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:13.762604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.762582    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:51:13.800146 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:13.800118    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11f34d2_0d88_4cb5_9df7_31f29d0dd5b0.slice/crio-0e786509ad9acd421e063570ecd22508314b8c0d14d65ef1fac987619cbbc4dd WatchSource:0}: Error finding container 0e786509ad9acd421e063570ecd22508314b8c0d14d65ef1fac987619cbbc4dd: Status 404 returned error can't find the container with id 0e786509ad9acd421e063570ecd22508314b8c0d14d65ef1fac987619cbbc4dd
Nov 02 21:51:13.803343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.803322    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"]
Nov 02 21:51:13.814990 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.814971    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-certs\") pod \"6bf178a9-eb59-4865-be00-c22072fe8439\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") "
Nov 02 21:51:13.815077 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815032    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-istio-token\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-istio-token\") pod \"6bf178a9-eb59-4865-be00-c22072fe8439\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") "
Nov 02 21:51:13.815077 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815052    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-socket\") pod \"6bf178a9-eb59-4865-be00-c22072fe8439\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") "
Nov 02 21:51:13.815212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815101    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-kube-api-access-wvpn8\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-kube-api-access-wvpn8\") pod \"6bf178a9-eb59-4865-be00-c22072fe8439\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") "
Nov 02 21:51:13.815212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815173    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf178a9-eb59-4865-be00-c22072fe8439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf178a9-eb59-4865-be00-c22072fe8439-istiod-ca-cert\") pod \"6bf178a9-eb59-4865-be00-c22072fe8439\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") "
Nov 02 21:51:13.815212 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815202    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-data\") pod \"6bf178a9-eb59-4865-be00-c22072fe8439\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") "
Nov 02 21:51:13.815368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815255    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-credential-socket\") pod \"6bf178a9-eb59-4865-be00-c22072fe8439\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") "
Nov 02 21:51:13.815368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815274    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-certs" pod "6bf178a9-eb59-4865-be00-c22072fe8439" (UID: "6bf178a9-eb59-4865-be00-c22072fe8439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:51:13.815368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815333    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bf178a9-eb59-4865-be00-c22072fe8439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf178a9-eb59-4865-be00-c22072fe8439-istio-podinfo\") pod \"6bf178a9-eb59-4865-be00-c22072fe8439\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") "
Nov 02 21:51:13.815534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815372    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-socket" pod "6bf178a9-eb59-4865-be00-c22072fe8439" (UID: "6bf178a9-eb59-4865-be00-c22072fe8439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:51:13.815534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815379    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-envoy\") pod \"6bf178a9-eb59-4865-be00-c22072fe8439\" (UID: \"6bf178a9-eb59-4865-be00-c22072fe8439\") "
Nov 02 21:51:13.815647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815559    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-credential-socket" pod "6bf178a9-eb59-4865-be00-c22072fe8439" (UID: "6bf178a9-eb59-4865-be00-c22072fe8439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:51:13.815647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815601    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf178a9-eb59-4865-be00-c22072fe8439-istiod-ca-cert" pod "6bf178a9-eb59-4865-be00-c22072fe8439" (UID: "6bf178a9-eb59-4865-be00-c22072fe8439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:51:13.815773 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815672    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-data" pod "6bf178a9-eb59-4865-be00-c22072fe8439" (UID: "6bf178a9-eb59-4865-be00-c22072fe8439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:51:13.815976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.815956    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:13.816142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.816126    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf178a9-eb59-4865-be00-c22072fe8439-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:13.816267 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.816247    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:13.816374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.816269    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:13.816374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.816283    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:13.817715 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.817691    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-envoy" pod "6bf178a9-eb59-4865-be00-c22072fe8439" (UID: "6bf178a9-eb59-4865-be00-c22072fe8439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:51:13.817899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.817876    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-istio-token" pod "6bf178a9-eb59-4865-be00-c22072fe8439" (UID: "6bf178a9-eb59-4865-be00-c22072fe8439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:51:13.817983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.817917    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-kube-api-access-wvpn8" pod "6bf178a9-eb59-4865-be00-c22072fe8439" (UID: "6bf178a9-eb59-4865-be00-c22072fe8439"). InnerVolumeSpecName "kube-api-access-wvpn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:51:13.818039 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.818025    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bf178a9-eb59-4865-be00-c22072fe8439-istio-podinfo" pod "6bf178a9-eb59-4865-be00-c22072fe8439" (UID: "6bf178a9-eb59-4865-be00-c22072fe8439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:51:13.917511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.917438    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:13.917511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.917465    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvpn8\" (UniqueName: \"kubernetes.io/projected/6bf178a9-eb59-4865-be00-c22072fe8439-kube-api-access-wvpn8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:13.917511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.917476    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf178a9-eb59-4865-be00-c22072fe8439-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:13.917511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:13.917486    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf178a9-eb59-4865-be00-c22072fe8439-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:14.116471 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:51:14.116442    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:51:14.214702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.214621    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr" event={"ID":"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0","Type":"ContainerStarted","Data":"497f4a45516e9490b272fc2645fec2a49d03590deade5adb138056fe5e91d4ae"}
Nov 02 21:51:14.214702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.214682    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr" event={"ID":"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0","Type":"ContainerStarted","Data":"0e786509ad9acd421e063570ecd22508314b8c0d14d65ef1fac987619cbbc4dd"}
Nov 02 21:51:14.216374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.216349    4528 generic.go:356] "Generic (PLEG): container finished" podID="6bf178a9-eb59-4865-be00-c22072fe8439" containerID="6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88" exitCode=0
Nov 02 21:51:14.216529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.216390    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"
Nov 02 21:51:14.216529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.216437    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" event={"ID":"6bf178a9-eb59-4865-be00-c22072fe8439","Type":"ContainerDied","Data":"6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88"}
Nov 02 21:51:14.216529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.216493    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4" event={"ID":"6bf178a9-eb59-4865-be00-c22072fe8439","Type":"ContainerDied","Data":"7e8a78884a79e8385330f77665d6cf2b26daf4127896afe37597e6ec4937b84d"}
Nov 02 21:51:14.216529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.216522    4528 scope.go:122] "RemoveContainer" containerID="6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88"
Nov 02 21:51:14.226121 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.226097    4528 scope.go:122] "RemoveContainer" containerID="ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf"
Nov 02 21:51:14.234301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.234285    4528 scope.go:122] "RemoveContainer" containerID="6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88"
Nov 02 21:51:14.234570 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:51:14.234552    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88\": container with ID starting with 6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88 not found: ID does not exist" containerID="6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88"
Nov 02 21:51:14.234624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.234579    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88"} err="failed to get container status \"6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88\": rpc error: code = NotFound desc = could not find container \"6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88\": container with ID starting with 6649814542a191589b3c5247c1d08bf3129b78ee05556f04c2475a589a806e88 not found: ID does not exist"
Nov 02 21:51:14.234624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.234596    4528 scope.go:122] "RemoveContainer" containerID="ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf"
Nov 02 21:51:14.234821 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:51:14.234804    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf\": container with ID starting with ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf not found: ID does not exist" containerID="ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf"
Nov 02 21:51:14.234866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.234827    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf"} err="failed to get container status \"ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf\": rpc error: code = NotFound desc = could not find container \"ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf\": container with ID starting with ba4efd55e7eef9d2cf00aba3fab0bd995d3f6db7ece00b0851672ce8887a7cbf not found: ID does not exist"
Nov 02 21:51:14.330544 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.330496    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr" podStartSLOduration=1.33048293 podStartE2EDuration="1.33048293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:14.328971888 +0000 UTC m=+2561.592508205" watchObservedRunningTime="2026-11-02 21:51:14.33048293 +0000 UTC m=+2561.594019247"
Nov 02 21:51:14.444193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.444102    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"]
Nov 02 21:51:14.451946 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.451916    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-8e90da-openshift-default-54696b8c49-p7fn4"]
Nov 02 21:51:14.913563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:14.913530    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" path="/var/lib/kubelet/pods/6bf178a9-eb59-4865-be00-c22072fe8439/volumes"
Nov 02 21:51:15.223487 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:15.223449    4528 generic.go:356] "Generic (PLEG): container finished" podID="f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0" containerID="497f4a45516e9490b272fc2645fec2a49d03590deade5adb138056fe5e91d4ae" exitCode=0
Nov 02 21:51:15.223962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:15.223500    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr" event={"ID":"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0","Type":"ContainerDied","Data":"497f4a45516e9490b272fc2645fec2a49d03590deade5adb138056fe5e91d4ae"}
Nov 02 21:51:15.749871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:15.749840    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"]
Nov 02 21:51:16.112577 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.112547    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"]
Nov 02 21:51:16.113062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.113048    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy"
Nov 02 21:51:16.113121 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.113063    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy"
Nov 02 21:51:16.113201 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.113191    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy"
Nov 02 21:51:16.113234 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.113204    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy"
Nov 02 21:51:16.116787 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.116767    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.156846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.156802    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.180158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.180128    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.180605 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.180511    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.181754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.181721    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.181921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.181792    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.181921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.181859    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.181921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.181889    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpbz\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-kube-api-access-xdpbz\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.182294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.181950    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.182294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.181974    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.182294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.179472    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0c9014-openshift-default-dockercfg-7ll9g\""
Nov 02 21:51:16.191207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.189075    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"]
Nov 02 21:51:16.283553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.283522    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.283563    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.283625    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.283643    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.283678    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.283746    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.283785    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpbz\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-kube-api-access-xdpbz\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.283879    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.283903    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.284034    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.284087    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.284466    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284735 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.284708    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.284947 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.284919    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.287541 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.287521    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.287749 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.287730    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.341853 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.341823    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.342658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.342626    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpbz\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-kube-api-access-xdpbz\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.385558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.385531    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:16.431101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.431076    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:16.486246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.485668    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-xks-ca\") pod \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") "
Nov 02 21:51:16.486246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.485716    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-kube-api-access-78rsk\" (UniqueName: \"kubernetes.io/projected/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-kube-api-access-78rsk\") pod \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") "
Nov 02 21:51:16.486246 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.485811    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-openshift-ca\") pod \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\" (UID: \"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0\") "
Nov 02 21:51:16.486496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.486330    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-openshift-ca" pod "f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0" (UID: "f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:51:16.487991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.487965    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-xks-ca" pod "f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0" (UID: "f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:51:16.488465 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.488441    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-kube-api-access-78rsk" pod "f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0" (UID: "f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0"). InnerVolumeSpecName "kube-api-access-78rsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:51:16.587055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.587024    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:16.587055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.587050    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78rsk\" (UniqueName: \"kubernetes.io/projected/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-kube-api-access-78rsk\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:16.587055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.587060    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:16.849520 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:16.849484    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49db25d8_09d9_45be_8fc7_5482fb7a9fbb.slice/crio-c59e99175a6f342ed6641acf5c5037eec1f6d511cbbd0f9de814640e16e52e72 WatchSource:0}: Error finding container c59e99175a6f342ed6641acf5c5037eec1f6d511cbbd0f9de814640e16e52e72: Status 404 returned error can't find the container with id c59e99175a6f342ed6641acf5c5037eec1f6d511cbbd0f9de814640e16e52e72
Nov 02 21:51:16.852160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.852120    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:16.852284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.852200    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:16.852284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.852233    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:16.852496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:16.852475    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"]
Nov 02 21:51:17.238810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.238656    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"
Nov 02 21:51:17.238810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.238689    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr" event={"ID":"f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0","Type":"ContainerDied","Data":"0e786509ad9acd421e063570ecd22508314b8c0d14d65ef1fac987619cbbc4dd"}
Nov 02 21:51:17.238810 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.238739    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e786509ad9acd421e063570ecd22508314b8c0d14d65ef1fac987619cbbc4dd"
Nov 02 21:51:17.240646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.240619    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" event={"ID":"49db25d8-09d9-45be-8fc7-5482fb7a9fbb","Type":"ContainerStarted","Data":"59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f"}
Nov 02 21:51:17.240783 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.240659    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" event={"ID":"49db25d8-09d9-45be-8fc7-5482fb7a9fbb","Type":"ContainerStarted","Data":"c59e99175a6f342ed6641acf5c5037eec1f6d511cbbd0f9de814640e16e52e72"}
Nov 02 21:51:17.417404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.417341    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podStartSLOduration=1.417321193 podStartE2EDuration="1.417321193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:17.366419748 +0000 UTC m=+2564.629956075" watchObservedRunningTime="2026-11-02 21:51:17.417321193 +0000 UTC m=+2564.680857511"
Nov 02 21:51:17.432308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.432278    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:17.433736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.433717    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 21:51:17.433806 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.433761    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 21:51:17.775409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.771748    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"]
Nov 02 21:51:17.775409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.772877    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy"
Nov 02 21:51:17.775409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.772899    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf178a9-eb59-4865-be00-c22072fe8439" containerName="istio-proxy"
Nov 02 21:51:17.775409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.772944    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0" containerName="revoke-keys"
Nov 02 21:51:17.775409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.772952    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0" containerName="revoke-keys"
Nov 02 21:51:17.775409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.773226    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0" containerName="revoke-keys"
Nov 02 21:51:17.779339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.779282    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:17.834091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.833952    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"]
Nov 02 21:51:17.834920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.834899    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:51:17.835011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.834998    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:51:17.835132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.835099    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:51:17.835185 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.835164    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:51:17.926852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.926801    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008260e5-f324-4292-8a55-1872010e6b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f4cd4d-xxgsc\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:17.927041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.926966    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008260e5-f324-4292-8a55-1872010e6b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f4cd4d-xxgsc\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:17.927041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:17.927018    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zspxh\" (UniqueName: \"kubernetes.io/projected/008260e5-f324-4292-8a55-1872010e6b36-kube-api-access-zspxh\") pod \"maas-api-revoke-sub-f0f4cd4d-xxgsc\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:18.028187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.028061    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008260e5-f324-4292-8a55-1872010e6b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f4cd4d-xxgsc\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:18.028187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.028172    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zspxh\" (UniqueName: \"kubernetes.io/projected/008260e5-f324-4292-8a55-1872010e6b36-kube-api-access-zspxh\") pod \"maas-api-revoke-sub-f0f4cd4d-xxgsc\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:18.028435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.028285    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008260e5-f324-4292-8a55-1872010e6b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f4cd4d-xxgsc\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:18.028849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.028817    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008260e5-f324-4292-8a55-1872010e6b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f4cd4d-xxgsc\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:18.030941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.030917    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008260e5-f324-4292-8a55-1872010e6b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f4cd4d-xxgsc\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:18.085035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.085007    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zspxh\" (UniqueName: \"kubernetes.io/projected/008260e5-f324-4292-8a55-1872010e6b36-kube-api-access-zspxh\") pod \"maas-api-revoke-sub-f0f4cd4d-xxgsc\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:18.095295 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.095270    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:18.432262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.432168    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 21:51:18.432262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.432222    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 21:51:18.488335 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:18.488303    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008260e5_f324_4292_8a55_1872010e6b36.slice/crio-2e7bc6a481ba2adff9d0b081e7ba77fe9fb74102a0cdf5725c60de3120ba4c0e WatchSource:0}: Error finding container 2e7bc6a481ba2adff9d0b081e7ba77fe9fb74102a0cdf5725c60de3120ba4c0e: Status 404 returned error can't find the container with id 2e7bc6a481ba2adff9d0b081e7ba77fe9fb74102a0cdf5725c60de3120ba4c0e
Nov 02 21:51:18.490406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:18.490380    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"]
Nov 02 21:51:19.255164 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:19.255098    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc" event={"ID":"008260e5-f324-4292-8a55-1872010e6b36","Type":"ContainerStarted","Data":"ec4b7825f8dd8df4683a630ef74cfab02854d81f35b6be4a0ca52268252a1612"}
Nov 02 21:51:19.255339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:19.255193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc" event={"ID":"008260e5-f324-4292-8a55-1872010e6b36","Type":"ContainerStarted","Data":"2e7bc6a481ba2adff9d0b081e7ba77fe9fb74102a0cdf5725c60de3120ba4c0e"}
Nov 02 21:51:19.388413 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:19.388367    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc" podStartSLOduration=2.388353528 podStartE2EDuration="2.388353528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:19.385866416 +0000 UTC m=+2566.649402727" watchObservedRunningTime="2026-11-02 21:51:19.388353528 +0000 UTC m=+2566.651889845"
Nov 02 21:51:19.431686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:19.431655    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 21:51:19.431871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:19.431709    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 21:51:20.262844 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:20.262808    4528 generic.go:356] "Generic (PLEG): container finished" podID="008260e5-f324-4292-8a55-1872010e6b36" containerID="ec4b7825f8dd8df4683a630ef74cfab02854d81f35b6be4a0ca52268252a1612" exitCode=0
Nov 02 21:51:20.263252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:20.262882    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc" event={"ID":"008260e5-f324-4292-8a55-1872010e6b36","Type":"ContainerDied","Data":"ec4b7825f8dd8df4683a630ef74cfab02854d81f35b6be4a0ca52268252a1612"}
Nov 02 21:51:20.431508 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:20.431467    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 21:51:20.431678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:20.431522    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 21:51:20.750873 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:20.750786    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79" podUID="58e0764a-0f38-4904-ad72-f6c2c51c6840" containerName="payload-processing" containerID="cri-o://59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973" gracePeriod=25
Nov 02 21:51:21.056236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.056215    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:51:21.164637 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.164605    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e0764a-0f38-4904-ad72-f6c2c51c6840-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e0764a-0f38-4904-ad72-f6c2c51c6840-plugins-config-volume\") pod \"58e0764a-0f38-4904-ad72-f6c2c51c6840\" (UID: \"58e0764a-0f38-4904-ad72-f6c2c51c6840\") "
Nov 02 21:51:21.164836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.164693    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e0764a-0f38-4904-ad72-f6c2c51c6840-kube-api-access-jdmmz\" (UniqueName: \"kubernetes.io/projected/58e0764a-0f38-4904-ad72-f6c2c51c6840-kube-api-access-jdmmz\") pod \"58e0764a-0f38-4904-ad72-f6c2c51c6840\" (UID: \"58e0764a-0f38-4904-ad72-f6c2c51c6840\") "
Nov 02 21:51:21.164962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.164937    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e0764a-0f38-4904-ad72-f6c2c51c6840-plugins-config-volume" pod "58e0764a-0f38-4904-ad72-f6c2c51c6840" (UID: "58e0764a-0f38-4904-ad72-f6c2c51c6840"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:51:21.166748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.166725    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e0764a-0f38-4904-ad72-f6c2c51c6840-kube-api-access-jdmmz" pod "58e0764a-0f38-4904-ad72-f6c2c51c6840" (UID: "58e0764a-0f38-4904-ad72-f6c2c51c6840"). InnerVolumeSpecName "kube-api-access-jdmmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:51:21.266153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.266088    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e0764a-0f38-4904-ad72-f6c2c51c6840-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:21.266153 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.266154    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdmmz\" (UniqueName: \"kubernetes.io/projected/58e0764a-0f38-4904-ad72-f6c2c51c6840-kube-api-access-jdmmz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:21.270648 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.270614    4528 generic.go:356] "Generic (PLEG): container finished" podID="58e0764a-0f38-4904-ad72-f6c2c51c6840" containerID="59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973" exitCode=0
Nov 02 21:51:21.270770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.270662    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"
Nov 02 21:51:21.270770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.270698    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79" event={"ID":"58e0764a-0f38-4904-ad72-f6c2c51c6840","Type":"ContainerDied","Data":"59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973"}
Nov 02 21:51:21.270840 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.270779    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79" event={"ID":"58e0764a-0f38-4904-ad72-f6c2c51c6840","Type":"ContainerDied","Data":"3bcfa2231332f7a5dc6edad065deed7f8986587ca6405b892bcae0ffca13203f"}
Nov 02 21:51:21.270840 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.270805    4528 scope.go:122] "RemoveContainer" containerID="59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973"
Nov 02 21:51:21.282417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.282399    4528 scope.go:122] "RemoveContainer" containerID="59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973"
Nov 02 21:51:21.282700 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:51:21.282682    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973\": container with ID starting with 59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973 not found: ID does not exist" containerID="59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973"
Nov 02 21:51:21.282768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.282706    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973"} err="failed to get container status \"59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973\": rpc error: code = NotFound desc = could not find container \"59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973\": container with ID starting with 59a13781e981d66643fd459274746820c5e112d60b483464101881eddbd6e973 not found: ID does not exist"
Nov 02 21:51:21.432037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.431999    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 21:51:21.432233 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.432063    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 21:51:21.445836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.445807    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"]
Nov 02 21:51:21.448414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.448385    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-942164-85d59ffd89-8cr79"]
Nov 02 21:51:21.455671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.455648    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:21.569204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.569092    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008260e5-f324-4292-8a55-1872010e6b36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008260e5-f324-4292-8a55-1872010e6b36-maas-api-xks-ca\") pod \"008260e5-f324-4292-8a55-1872010e6b36\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") "
Nov 02 21:51:21.569368 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.569269    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008260e5-f324-4292-8a55-1872010e6b36-kube-api-access-zspxh\" (UniqueName: \"kubernetes.io/projected/008260e5-f324-4292-8a55-1872010e6b36-kube-api-access-zspxh\") pod \"008260e5-f324-4292-8a55-1872010e6b36\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") "
Nov 02 21:51:21.569415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.569401    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/008260e5-f324-4292-8a55-1872010e6b36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008260e5-f324-4292-8a55-1872010e6b36-maas-api-openshift-ca\") pod \"008260e5-f324-4292-8a55-1872010e6b36\" (UID: \"008260e5-f324-4292-8a55-1872010e6b36\") "
Nov 02 21:51:21.569723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.569693    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008260e5-f324-4292-8a55-1872010e6b36-maas-api-openshift-ca" pod "008260e5-f324-4292-8a55-1872010e6b36" (UID: "008260e5-f324-4292-8a55-1872010e6b36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:51:21.571173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.571152    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008260e5-f324-4292-8a55-1872010e6b36-maas-api-xks-ca" pod "008260e5-f324-4292-8a55-1872010e6b36" (UID: "008260e5-f324-4292-8a55-1872010e6b36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:51:21.571292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.571275    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008260e5-f324-4292-8a55-1872010e6b36-kube-api-access-zspxh" pod "008260e5-f324-4292-8a55-1872010e6b36" (UID: "008260e5-f324-4292-8a55-1872010e6b36"). InnerVolumeSpecName "kube-api-access-zspxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:51:21.670547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.670515    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/008260e5-f324-4292-8a55-1872010e6b36-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:21.670547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.670546    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zspxh\" (UniqueName: \"kubernetes.io/projected/008260e5-f324-4292-8a55-1872010e6b36-kube-api-access-zspxh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:21.670752 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:21.670556    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/008260e5-f324-4292-8a55-1872010e6b36-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:22.279228 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:22.279186    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc" event={"ID":"008260e5-f324-4292-8a55-1872010e6b36","Type":"ContainerDied","Data":"2e7bc6a481ba2adff9d0b081e7ba77fe9fb74102a0cdf5725c60de3120ba4c0e"}
Nov 02 21:51:22.279228 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:22.279218    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"
Nov 02 21:51:22.279680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:22.279247    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e7bc6a481ba2adff9d0b081e7ba77fe9fb74102a0cdf5725c60de3120ba4c0e"
Nov 02 21:51:22.431679 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:22.431644    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 21:51:22.431852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:22.431696    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 21:51:22.932420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:22.932382    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e0764a-0f38-4904-ad72-f6c2c51c6840" path="/var/lib/kubelet/pods/58e0764a-0f38-4904-ad72-f6c2c51c6840/volumes"
Nov 02 21:51:23.432198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:23.432164    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 21:51:23.432683 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:23.432221    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 21:51:24.468746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:24.468703    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:24.469952 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:24.469918    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:24.470467 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:24.470448    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 21:51:31.086914 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.086882    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"]
Nov 02 21:51:31.087415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.087376    4528 cpu_manager.go:414] "containerMap: removing container" podUID="008260e5-f324-4292-8a55-1872010e6b36" containerName="revoke-keys"
Nov 02 21:51:31.087415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.087389    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008260e5-f324-4292-8a55-1872010e6b36" containerName="revoke-keys"
Nov 02 21:51:31.087415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.087400    4528 cpu_manager.go:414] "containerMap: removing container" podUID="58e0764a-0f38-4904-ad72-f6c2c51c6840" containerName="payload-processing"
Nov 02 21:51:31.087415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.087405    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e0764a-0f38-4904-ad72-f6c2c51c6840" containerName="payload-processing"
Nov 02 21:51:31.087553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.087487    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="008260e5-f324-4292-8a55-1872010e6b36" containerName="revoke-keys"
Nov 02 21:51:31.087553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.087497    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e0764a-0f38-4904-ad72-f6c2c51c6840" containerName="payload-processing"
Nov 02 21:51:31.090719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.090701    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.146975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.144251    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"]
Nov 02 21:51:31.168131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.164092    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.185907    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krdgg\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-kube-api-access-krdgg\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.185973    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8332a66-8360-47eb-83d5-a8059cc2875b-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186011    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdr9\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-kube-api-access-dvdr9\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186042    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186071    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-envoy\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186146    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-credential-socket\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186184    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-podinfo\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186231    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186295    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istiod-ca-cert\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186332    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186367    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186413    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-certs\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186447    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186471    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-socket\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186537    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186573    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-data\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186601    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.189336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.186633    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-token\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.198836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.198800    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"]
Nov 02 21:51:31.202831 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.199728    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3c7706-openshift-default-dockercfg-dq5x6\""
Nov 02 21:51:31.288165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.286189    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"]
Nov 02 21:51:31.293932 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.293885    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.294239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.294220    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-data\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.294387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.294373    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.294535 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.294513    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-token\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.294675 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.294658    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdgg\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-kube-api-access-krdgg\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.294815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.294800    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8332a66-8360-47eb-83d5-a8059cc2875b-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.294923 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.294913    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdr9\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-kube-api-access-dvdr9\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.295006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.294997    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295071    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-envoy\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295172    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-credential-socket\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295211    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-podinfo\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295263    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295330    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istiod-ca-cert\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295410    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295449    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-certs\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295484    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.295622 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295511    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-socket\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.296280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295777    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8332a66-8360-47eb-83d5-a8059cc2875b-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.296280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.295870    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-data\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.296280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.296219    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.296450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.296302    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.296504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.296469    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.297423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.296811    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-credential-socket\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.297423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.296970    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istiod-ca-cert\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.297423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.297390    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-certs\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.297842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.297809    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-socket\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.299841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.299777    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-podinfo\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.301859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.301383    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-envoy\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.301859 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.301819    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.302498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.302471    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.355292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.355211    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-token\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.358258 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.358225    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdr9\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-kube-api-access-dvdr9\") pod \"e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") " pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.358457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.358441    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krdgg\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-kube-api-access-krdgg\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.359154 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.359138    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.410539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.410513    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:31.500049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.500018    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:31.847726 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:31.847692    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a95916_ee75_44bc_be9e_8cc8baed5bf1.slice/crio-af3286bf4756fef8a0407a09e60157489ee61009895656170fad1017d370ef12 WatchSource:0}: Error finding container af3286bf4756fef8a0407a09e60157489ee61009895656170fad1017d370ef12: Status 404 returned error can't find the container with id af3286bf4756fef8a0407a09e60157489ee61009895656170fad1017d370ef12
Nov 02 21:51:31.850471 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.850436    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:31.850593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.850514    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:31.850593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.850559    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:31.851351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.851331    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"]
Nov 02 21:51:31.898170 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:31.898139    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8332a66_8360_47eb_83d5_a8059cc2875b.slice/crio-69b214116b50b21e8f0cd1d5b69095dfebba85a8d68833c3634087461796ef44 WatchSource:0}: Error finding container 69b214116b50b21e8f0cd1d5b69095dfebba85a8d68833c3634087461796ef44: Status 404 returned error can't find the container with id 69b214116b50b21e8f0cd1d5b69095dfebba85a8d68833c3634087461796ef44
Nov 02 21:51:31.900687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.900656    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:31.900797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.900716    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:31.900797 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.900745    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:51:31.901346 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:31.901328    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"]
Nov 02 21:51:32.346301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.346259    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" event={"ID":"e8332a66-8360-47eb-83d5-a8059cc2875b","Type":"ContainerStarted","Data":"1f78bdb29269ad68601eaba931204cd4e95a1bbd6ce03676267197d41adc1571"}
Nov 02 21:51:32.346786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.346320    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" event={"ID":"e8332a66-8360-47eb-83d5-a8059cc2875b","Type":"ContainerStarted","Data":"69b214116b50b21e8f0cd1d5b69095dfebba85a8d68833c3634087461796ef44"}
Nov 02 21:51:32.349055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.348170    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" event={"ID":"c8a95916-ee75-44bc-be9e-8cc8baed5bf1","Type":"ContainerStarted","Data":"538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c"}
Nov 02 21:51:32.349055 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.348243    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" event={"ID":"c8a95916-ee75-44bc-be9e-8cc8baed5bf1","Type":"ContainerStarted","Data":"af3286bf4756fef8a0407a09e60157489ee61009895656170fad1017d370ef12"}
Nov 02 21:51:32.410852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.410808    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:32.416599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.416573    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:32.464384 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.464324    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podStartSLOduration=1.464311243 podStartE2EDuration="1.464311243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:32.463295317 +0000 UTC m=+2579.726831635" watchObservedRunningTime="2026-11-02 21:51:32.464311243 +0000 UTC m=+2579.727847616"
Nov 02 21:51:32.500499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.500468    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:32.509133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.508238    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 21:51:32.509133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.508313    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 21:51:32.580027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:32.579967    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" podStartSLOduration=1.579954581 podStartE2EDuration="1.579954581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:32.579778882 +0000 UTC m=+2579.843315198" watchObservedRunningTime="2026-11-02 21:51:32.579954581 +0000 UTC m=+2579.843490896"
Nov 02 21:51:33.354251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:33.354208    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:33.355049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:33.355031    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:51:33.501277 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:33.501244    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 21:51:33.512134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:33.501526    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 21:51:34.519133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:34.512198    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 21:51:34.519133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:34.512258    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 21:51:35.500960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:35.500923    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 21:51:35.501146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:35.500977    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 21:51:36.500858 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:36.500827    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 21:51:36.501308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:36.500876    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 21:51:37.247637 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.247600    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"]
Nov 02 21:51:37.252586 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.252566    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.334437 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.334389    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"]
Nov 02 21:51:37.371854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.371824    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:51:37.371854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.371824    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:51:37.371854 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.371837    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:51:37.372096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.371873    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:51:37.381637 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.381612    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf2a166-28r89\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.381769 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.381666    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s268c\" (UniqueName: \"kubernetes.io/projected/01cdaed5-23e4-43e2-9f09-11a4230b270c-kube-api-access-s268c\") pod \"maas-api-revoke-sub-fbf2a166-28r89\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.381807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.381779    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf2a166-28r89\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.482699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.482661    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf2a166-28r89\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.482893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.482726    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s268c\" (UniqueName: \"kubernetes.io/projected/01cdaed5-23e4-43e2-9f09-11a4230b270c-kube-api-access-s268c\") pod \"maas-api-revoke-sub-fbf2a166-28r89\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.482893 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.482749    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf2a166-28r89\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.483386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.483363    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf2a166-28r89\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.485393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.485374    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf2a166-28r89\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.500669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.500614    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 21:51:37.500669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.500659    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 21:51:37.539598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.539566    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s268c\" (UniqueName: \"kubernetes.io/projected/01cdaed5-23e4-43e2-9f09-11a4230b270c-kube-api-access-s268c\") pod \"maas-api-revoke-sub-fbf2a166-28r89\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.566353 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.566329    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:37.953729 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:37.953698    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cdaed5_23e4_43e2_9f09_11a4230b270c.slice/crio-4361ae78e619e27ccc0a612024e746c352cb7e583ca087ad2b1d1aa37653c7fb WatchSource:0}: Error finding container 4361ae78e619e27ccc0a612024e746c352cb7e583ca087ad2b1d1aa37653c7fb: Status 404 returned error can't find the container with id 4361ae78e619e27ccc0a612024e746c352cb7e583ca087ad2b1d1aa37653c7fb
Nov 02 21:51:37.957427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:37.957402    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"]
Nov 02 21:51:38.386265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:38.386229    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89" event={"ID":"01cdaed5-23e4-43e2-9f09-11a4230b270c","Type":"ContainerStarted","Data":"47df24c4b6cd6beb2c204839d392407da83cc05619bceb734726e910a9730a56"}
Nov 02 21:51:38.386453 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:38.386295    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89" event={"ID":"01cdaed5-23e4-43e2-9f09-11a4230b270c","Type":"ContainerStarted","Data":"4361ae78e619e27ccc0a612024e746c352cb7e583ca087ad2b1d1aa37653c7fb"}
Nov 02 21:51:38.501379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:38.501303    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 21:51:38.501379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:38.501356    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 21:51:38.502271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:38.502221    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89" podStartSLOduration=1.502209135 podStartE2EDuration="1.502209135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:38.498223629 +0000 UTC m=+2585.761759945" watchObservedRunningTime="2026-11-02 21:51:38.502209135 +0000 UTC m=+2585.765745451"
Nov 02 21:51:39.227179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.225955    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"]
Nov 02 21:51:39.238354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.230866    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:39.289142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.284212    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3c7706\""
Nov 02 21:51:39.291251 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.289583    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"]
Nov 02 21:51:39.323052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.323022    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3c7706-dockercfg-w5nvg\""
Nov 02 21:51:39.326638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.326614    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-kube-api-access-tz7xm\") pod \"payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5\" (UID: \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:39.326767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.326683    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5\" (UID: \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:39.373527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.373497    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"]
Nov 02 21:51:39.377386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.377368    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:39.394788 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.394764    4528 generic.go:356] "Generic (PLEG): container finished" podID="01cdaed5-23e4-43e2-9f09-11a4230b270c" containerID="47df24c4b6cd6beb2c204839d392407da83cc05619bceb734726e910a9730a56" exitCode=0
Nov 02 21:51:39.394910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.394795    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89" event={"ID":"01cdaed5-23e4-43e2-9f09-11a4230b270c","Type":"ContainerDied","Data":"47df24c4b6cd6beb2c204839d392407da83cc05619bceb734726e910a9730a56"}
Nov 02 21:51:39.428635 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.428594    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-kube-api-access-tz7xm\") pod \"payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5\" (UID: \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:39.428772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.428720    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5\" (UID: \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:39.430478 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.429747    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5\" (UID: \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:39.430585 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.430540    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"]
Nov 02 21:51:39.485967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.485882    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-kube-api-access-tz7xm\") pod \"payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5\" (UID: \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:39.505331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.505306    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:39.505597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.505573    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:39.506255 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.506238    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 21:51:39.530590 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.530567    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee16119-ee33-4dd9-9764-465ad0c3c241-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv\" (UID: \"fee16119-ee33-4dd9-9764-465ad0c3c241\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:39.530736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.530718    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pjk7\" (UniqueName: \"kubernetes.io/projected/fee16119-ee33-4dd9-9764-465ad0c3c241-kube-api-access-9pjk7\") pod \"payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv\" (UID: \"fee16119-ee33-4dd9-9764-465ad0c3c241\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:39.588286 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.588264    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:39.632480 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.632440    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pjk7\" (UniqueName: \"kubernetes.io/projected/fee16119-ee33-4dd9-9764-465ad0c3c241-kube-api-access-9pjk7\") pod \"payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv\" (UID: \"fee16119-ee33-4dd9-9764-465ad0c3c241\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:39.632678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.632636    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee16119-ee33-4dd9-9764-465ad0c3c241-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv\" (UID: \"fee16119-ee33-4dd9-9764-465ad0c3c241\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:39.633428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.633403    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee16119-ee33-4dd9-9764-465ad0c3c241-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv\" (UID: \"fee16119-ee33-4dd9-9764-465ad0c3c241\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:39.689826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.689792    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pjk7\" (UniqueName: \"kubernetes.io/projected/fee16119-ee33-4dd9-9764-465ad0c3c241-kube-api-access-9pjk7\") pod \"payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv\" (UID: \"fee16119-ee33-4dd9-9764-465ad0c3c241\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:39.691545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.691515    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:39.954895 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:39.954845    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c85644c_b5a3_4845_9f93_6e1afbeb66ca.slice/crio-8d21d3f64502d09d2be01b473a13baf9185c352b85dd967e34d29bd30d0bf64b WatchSource:0}: Error finding container 8d21d3f64502d09d2be01b473a13baf9185c352b85dd967e34d29bd30d0bf64b: Status 404 returned error can't find the container with id 8d21d3f64502d09d2be01b473a13baf9185c352b85dd967e34d29bd30d0bf64b
Nov 02 21:51:39.958580 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:39.958558    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"]
Nov 02 21:51:40.088278 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:40.088240    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee16119_ee33_4dd9_9764_465ad0c3c241.slice/crio-3b9c689a6893388df176d8293597ff8c71c8774dacaeb9ef1d5d2d056f3c4fe6 WatchSource:0}: Error finding container 3b9c689a6893388df176d8293597ff8c71c8774dacaeb9ef1d5d2d056f3c4fe6: Status 404 returned error can't find the container with id 3b9c689a6893388df176d8293597ff8c71c8774dacaeb9ef1d5d2d056f3c4fe6
Nov 02 21:51:40.091953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.091929    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"]
Nov 02 21:51:40.403053 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.402951    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv" event={"ID":"fee16119-ee33-4dd9-9764-465ad0c3c241","Type":"ContainerStarted","Data":"d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3"}
Nov 02 21:51:40.403053 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.403048    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv" event={"ID":"fee16119-ee33-4dd9-9764-465ad0c3c241","Type":"ContainerStarted","Data":"3b9c689a6893388df176d8293597ff8c71c8774dacaeb9ef1d5d2d056f3c4fe6"}
Nov 02 21:51:40.403578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.403081    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:40.404673 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.404641    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5" event={"ID":"2c85644c-b5a3-4845-9f93-6e1afbeb66ca","Type":"ContainerStarted","Data":"262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96"}
Nov 02 21:51:40.405002 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.404901    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:40.405002 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.404949    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5" event={"ID":"2c85644c-b5a3-4845-9f93-6e1afbeb66ca","Type":"ContainerStarted","Data":"8d21d3f64502d09d2be01b473a13baf9185c352b85dd967e34d29bd30d0bf64b"}
Nov 02 21:51:40.519098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.519049    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv" podStartSLOduration=1.5190358229999998 podStartE2EDuration="1.519035823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:40.518742683 +0000 UTC m=+2587.782278999" watchObservedRunningTime="2026-11-02 21:51:40.519035823 +0000 UTC m=+2587.782572139"
Nov 02 21:51:40.549568 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.549545    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:40.643837 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.643772    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01cdaed5-23e4-43e2-9f09-11a4230b270c-kube-api-access-s268c\" (UniqueName: \"kubernetes.io/projected/01cdaed5-23e4-43e2-9f09-11a4230b270c-kube-api-access-s268c\") pod \"01cdaed5-23e4-43e2-9f09-11a4230b270c\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") "
Nov 02 21:51:40.644219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.644158    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-openshift-ca\") pod \"01cdaed5-23e4-43e2-9f09-11a4230b270c\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") "
Nov 02 21:51:40.644538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.644466    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-xks-ca\") pod \"01cdaed5-23e4-43e2-9f09-11a4230b270c\" (UID: \"01cdaed5-23e4-43e2-9f09-11a4230b270c\") "
Nov 02 21:51:40.647346 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.647283    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-openshift-ca" pod "01cdaed5-23e4-43e2-9f09-11a4230b270c" (UID: "01cdaed5-23e4-43e2-9f09-11a4230b270c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:51:40.664668 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.656495    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5" podStartSLOduration=1.656473061 podStartE2EDuration="1.656473061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:40.635201075 +0000 UTC m=+2587.898737396" watchObservedRunningTime="2026-11-02 21:51:40.656473061 +0000 UTC m=+2587.920009379"
Nov 02 21:51:40.669712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.668770    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-xks-ca" pod "01cdaed5-23e4-43e2-9f09-11a4230b270c" (UID: "01cdaed5-23e4-43e2-9f09-11a4230b270c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:51:40.669712 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.669645    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cdaed5-23e4-43e2-9f09-11a4230b270c-kube-api-access-s268c" pod "01cdaed5-23e4-43e2-9f09-11a4230b270c" (UID: "01cdaed5-23e4-43e2-9f09-11a4230b270c"). InnerVolumeSpecName "kube-api-access-s268c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:51:40.746795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.746765    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s268c\" (UniqueName: \"kubernetes.io/projected/01cdaed5-23e4-43e2-9f09-11a4230b270c-kube-api-access-s268c\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:40.746795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.746794    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:40.746983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.746808    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01cdaed5-23e4-43e2-9f09-11a4230b270c-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:40.996974 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:40.996944    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:51:41.412912 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:41.412875    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"
Nov 02 21:51:41.413404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:41.412932    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89" event={"ID":"01cdaed5-23e4-43e2-9f09-11a4230b270c","Type":"ContainerDied","Data":"4361ae78e619e27ccc0a612024e746c352cb7e583ca087ad2b1d1aa37653c7fb"}
Nov 02 21:51:41.413404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:41.412978    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4361ae78e619e27ccc0a612024e746c352cb7e583ca087ad2b1d1aa37653c7fb"
Nov 02 21:51:44.113376 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:51:44.113346    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:51:44.718389 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.718359    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"]
Nov 02 21:51:44.718901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.718888    4528 cpu_manager.go:414] "containerMap: removing container" podUID="01cdaed5-23e4-43e2-9f09-11a4230b270c" containerName="revoke-keys"
Nov 02 21:51:44.718947 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.718902    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cdaed5-23e4-43e2-9f09-11a4230b270c" containerName="revoke-keys"
Nov 02 21:51:44.718998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.718985    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cdaed5-23e4-43e2-9f09-11a4230b270c" containerName="revoke-keys"
Nov 02 21:51:44.722443 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.722428    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:44.775170 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.775143    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"]
Nov 02 21:51:44.822034 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.821998    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:51:44.822034 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.822011    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:51:44.822269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.822068    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:51:44.824703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.824682    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:51:44.851598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.851575    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10bcb25a-hz969\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:44.851704 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.851647    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10bcb25a-hz969\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:44.851805 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.851790    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hkc\" (UniqueName: \"kubernetes.io/projected/49825c7e-57c9-43c1-a274-f0f948561807-kube-api-access-52hkc\") pod \"maas-api-revoke-sub-10bcb25a-hz969\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:44.953259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.953226    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10bcb25a-hz969\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:44.953448 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.953284    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10bcb25a-hz969\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:44.953448 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.953369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hkc\" (UniqueName: \"kubernetes.io/projected/49825c7e-57c9-43c1-a274-f0f948561807-kube-api-access-52hkc\") pod \"maas-api-revoke-sub-10bcb25a-hz969\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:44.953913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.953891    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10bcb25a-hz969\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:44.955565 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:44.955539    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10bcb25a-hz969\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:45.009896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:45.009834    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hkc\" (UniqueName: \"kubernetes.io/projected/49825c7e-57c9-43c1-a274-f0f948561807-kube-api-access-52hkc\") pod \"maas-api-revoke-sub-10bcb25a-hz969\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:45.035393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:45.035369    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:45.403961 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:45.403918    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49825c7e_57c9_43c1_a274_f0f948561807.slice/crio-26ece33ad8f89614f7b41e216d8b55d6583993ddbd24ce6b7eff64fd8820ee49 WatchSource:0}: Error finding container 26ece33ad8f89614f7b41e216d8b55d6583993ddbd24ce6b7eff64fd8820ee49: Status 404 returned error can't find the container with id 26ece33ad8f89614f7b41e216d8b55d6583993ddbd24ce6b7eff64fd8820ee49
Nov 02 21:51:45.408146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:45.407517    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"]
Nov 02 21:51:45.500702 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:45.500675    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969" event={"ID":"49825c7e-57c9-43c1-a274-f0f948561807","Type":"ContainerStarted","Data":"26ece33ad8f89614f7b41e216d8b55d6583993ddbd24ce6b7eff64fd8820ee49"}
Nov 02 21:51:46.508254 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:46.508170    4528 generic.go:356] "Generic (PLEG): container finished" podID="49825c7e-57c9-43c1-a274-f0f948561807" containerID="5258b5995d7f91b7e7f689cc05c55ed85705c4409f8b9a42eda6f1ccd19ecfc9" exitCode=0
Nov 02 21:51:46.508610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:46.508260    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969" event={"ID":"49825c7e-57c9-43c1-a274-f0f948561807","Type":"ContainerDied","Data":"5258b5995d7f91b7e7f689cc05c55ed85705c4409f8b9a42eda6f1ccd19ecfc9"}
Nov 02 21:51:47.653675 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.653653    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:47.783397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.783328    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-openshift-ca\") pod \"49825c7e-57c9-43c1-a274-f0f948561807\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") "
Nov 02 21:51:47.783397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.783393    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49825c7e-57c9-43c1-a274-f0f948561807-kube-api-access-52hkc\" (UniqueName: \"kubernetes.io/projected/49825c7e-57c9-43c1-a274-f0f948561807-kube-api-access-52hkc\") pod \"49825c7e-57c9-43c1-a274-f0f948561807\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") "
Nov 02 21:51:47.783621 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.783446    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-xks-ca\") pod \"49825c7e-57c9-43c1-a274-f0f948561807\" (UID: \"49825c7e-57c9-43c1-a274-f0f948561807\") "
Nov 02 21:51:47.783727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.783705    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-openshift-ca" pod "49825c7e-57c9-43c1-a274-f0f948561807" (UID: "49825c7e-57c9-43c1-a274-f0f948561807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:51:47.784062 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.784036    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:47.785526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.785491    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-xks-ca" pod "49825c7e-57c9-43c1-a274-f0f948561807" (UID: "49825c7e-57c9-43c1-a274-f0f948561807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:51:47.785643 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.785592    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49825c7e-57c9-43c1-a274-f0f948561807-kube-api-access-52hkc" pod "49825c7e-57c9-43c1-a274-f0f948561807" (UID: "49825c7e-57c9-43c1-a274-f0f948561807"). InnerVolumeSpecName "kube-api-access-52hkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:51:47.885023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.884998    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52hkc\" (UniqueName: \"kubernetes.io/projected/49825c7e-57c9-43c1-a274-f0f948561807-kube-api-access-52hkc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:47.885023 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:47.885024    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49825c7e-57c9-43c1-a274-f0f948561807-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:48.521042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:48.521014    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"
Nov 02 21:51:48.521042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:48.521024    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969" event={"ID":"49825c7e-57c9-43c1-a274-f0f948561807","Type":"ContainerDied","Data":"26ece33ad8f89614f7b41e216d8b55d6583993ddbd24ce6b7eff64fd8820ee49"}
Nov 02 21:51:48.521257 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:48.521067    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26ece33ad8f89614f7b41e216d8b55d6583993ddbd24ce6b7eff64fd8820ee49"
Nov 02 21:51:51.416263 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:51.416233    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:51:51.416723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:51.416312    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:51:52.283874 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.283842    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"]
Nov 02 21:51:52.284351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.284338    4528 cpu_manager.go:414] "containerMap: removing container" podUID="49825c7e-57c9-43c1-a274-f0f948561807" containerName="revoke-keys"
Nov 02 21:51:52.284418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.284352    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49825c7e-57c9-43c1-a274-f0f948561807" containerName="revoke-keys"
Nov 02 21:51:52.284482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.284473    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="49825c7e-57c9-43c1-a274-f0f948561807" containerName="revoke-keys"
Nov 02 21:51:52.287977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.287963    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.339778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.339751    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"]
Nov 02 21:51:52.386623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.386592    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:51:52.386623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.386620    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:51:52.386818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.386628    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:51:52.390397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.390378    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:51:52.436228 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.436201    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0022265-vq7md\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.436554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.436311    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0022265-vq7md\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.436554 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.436344    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5gg\" (UniqueName: \"kubernetes.io/projected/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-kube-api-access-pz5gg\") pod \"maas-api-revoke-sub-e0022265-vq7md\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.537319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.537256    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0022265-vq7md\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.537456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.537322    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0022265-vq7md\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.537456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.537351    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5gg\" (UniqueName: \"kubernetes.io/projected/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-kube-api-access-pz5gg\") pod \"maas-api-revoke-sub-e0022265-vq7md\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.537967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.537941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0022265-vq7md\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.539656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.539634    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0022265-vq7md\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.593901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.593876    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5gg\" (UniqueName: \"kubernetes.io/projected/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-kube-api-access-pz5gg\") pod \"maas-api-revoke-sub-e0022265-vq7md\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.601725 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.601704    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:52.986253 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:51:52.986214    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a9c036_44f7_4fe0_9d54_2a4f3e32e1a7.slice/crio-998bd277e5e3bf9d87279228531acfbeed91df9cf8617c77f89e9cc67ec518b6 WatchSource:0}: Error finding container 998bd277e5e3bf9d87279228531acfbeed91df9cf8617c77f89e9cc67ec518b6: Status 404 returned error can't find the container with id 998bd277e5e3bf9d87279228531acfbeed91df9cf8617c77f89e9cc67ec518b6
Nov 02 21:51:52.990598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:52.990574    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"]
Nov 02 21:51:53.552259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:53.552220    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md" event={"ID":"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7","Type":"ContainerStarted","Data":"9f981e378657ea722c81cd54cab4fa3cc2c9f04fa03bbe8d37b8653ffdffae15"}
Nov 02 21:51:53.552642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:53.552319    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md" event={"ID":"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7","Type":"ContainerStarted","Data":"998bd277e5e3bf9d87279228531acfbeed91df9cf8617c77f89e9cc67ec518b6"}
Nov 02 21:51:53.670015 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:53.669898    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md" podStartSLOduration=1.6698793159999998 podStartE2EDuration="1.669879316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:53.668089628 +0000 UTC m=+2600.931625946" watchObservedRunningTime="2026-11-02 21:51:53.669879316 +0000 UTC m=+2600.933415634"
Nov 02 21:51:54.559898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:54.559866    4528 generic.go:356] "Generic (PLEG): container finished" podID="a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7" containerID="9f981e378657ea722c81cd54cab4fa3cc2c9f04fa03bbe8d37b8653ffdffae15" exitCode=0
Nov 02 21:51:54.560329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:54.559949    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md" event={"ID":"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7","Type":"ContainerDied","Data":"9f981e378657ea722c81cd54cab4fa3cc2c9f04fa03bbe8d37b8653ffdffae15"}
Nov 02 21:51:55.705802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.705776    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:55.806369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.806337    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-xks-ca\") pod \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") "
Nov 02 21:51:55.806521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.806383    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-openshift-ca\") pod \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") "
Nov 02 21:51:55.806521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.806414    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-kube-api-access-pz5gg\" (UniqueName: \"kubernetes.io/projected/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-kube-api-access-pz5gg\") pod \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\" (UID: \"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7\") "
Nov 02 21:51:55.806792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.806767    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-openshift-ca" pod "a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7" (UID: "a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:51:55.808461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.808433    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-kube-api-access-pz5gg" pod "a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7" (UID: "a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7"). InnerVolumeSpecName "kube-api-access-pz5gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:51:55.808563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.808483    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-xks-ca" pod "a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7" (UID: "a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:51:55.907774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.907698    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:55.907774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.907727    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:55.907774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:55.907739    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz5gg\" (UniqueName: \"kubernetes.io/projected/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7-kube-api-access-pz5gg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:51:56.572736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:56.572711    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"
Nov 02 21:51:56.572910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:56.572705    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md" event={"ID":"a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7","Type":"ContainerDied","Data":"998bd277e5e3bf9d87279228531acfbeed91df9cf8617c77f89e9cc67ec518b6"}
Nov 02 21:51:56.572910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:51:56.572832    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="998bd277e5e3bf9d87279228531acfbeed91df9cf8617c77f89e9cc67ec518b6"
Nov 02 21:52:00.062172 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.062142    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"]
Nov 02 21:52:00.062627 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.062614    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7" containerName="revoke-keys"
Nov 02 21:52:00.062627 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.062627    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7" containerName="revoke-keys"
Nov 02 21:52:00.062738 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.062728    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7" containerName="revoke-keys"
Nov 02 21:52:00.065852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.065836    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.117680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.117655    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"]
Nov 02 21:52:00.152101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.152079    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f6ccca7-7mpbg\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.152280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.152259    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f6ccca7-7mpbg\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.152399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.152378    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qlx4\" (UniqueName: \"kubernetes.io/projected/3c9ed414-d247-480a-82d4-56c3621a6b94-kube-api-access-4qlx4\") pod \"maas-api-revoke-sub-3f6ccca7-7mpbg\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.164265 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.164241    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:52:00.164377 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.164269    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:52:00.168279 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.168266    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:52:00.168358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.168298    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:52:00.252966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.252942    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f6ccca7-7mpbg\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.253092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.253039    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f6ccca7-7mpbg\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.253169 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.253094    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qlx4\" (UniqueName: \"kubernetes.io/projected/3c9ed414-d247-480a-82d4-56c3621a6b94-kube-api-access-4qlx4\") pod \"maas-api-revoke-sub-3f6ccca7-7mpbg\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.253570 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.253547    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f6ccca7-7mpbg\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.255387 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.255364    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f6ccca7-7mpbg\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.309599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.309567    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qlx4\" (UniqueName: \"kubernetes.io/projected/3c9ed414-d247-480a-82d4-56c3621a6b94-kube-api-access-4qlx4\") pod \"maas-api-revoke-sub-3f6ccca7-7mpbg\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.378931 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.378870    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:00.744998 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:00.744954    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9ed414_d247_480a_82d4_56c3621a6b94.slice/crio-f032622b5830de4afaf75e7263f073d5c933d608f70d72dbe08b0a9f83bf4b5f WatchSource:0}: Error finding container f032622b5830de4afaf75e7263f073d5c933d608f70d72dbe08b0a9f83bf4b5f: Status 404 returned error can't find the container with id f032622b5830de4afaf75e7263f073d5c933d608f70d72dbe08b0a9f83bf4b5f
Nov 02 21:52:00.748512 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:00.748490    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"]
Nov 02 21:52:01.633142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:01.633098    4528 generic.go:356] "Generic (PLEG): container finished" podID="3c9ed414-d247-480a-82d4-56c3621a6b94" containerID="04c25dc68b66f9e10c47737d021668afd1d55d12f20195a646b55e48367f9e5f" exitCode=0
Nov 02 21:52:01.633459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:01.633156    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg" event={"ID":"3c9ed414-d247-480a-82d4-56c3621a6b94","Type":"ContainerDied","Data":"04c25dc68b66f9e10c47737d021668afd1d55d12f20195a646b55e48367f9e5f"}
Nov 02 21:52:01.633459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:01.633201    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg" event={"ID":"3c9ed414-d247-480a-82d4-56c3621a6b94","Type":"ContainerStarted","Data":"f032622b5830de4afaf75e7263f073d5c933d608f70d72dbe08b0a9f83bf4b5f"}
Nov 02 21:52:02.786548 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.786518    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:02.885352 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.885323    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9ed414-d247-480a-82d4-56c3621a6b94-kube-api-access-4qlx4\" (UniqueName: \"kubernetes.io/projected/3c9ed414-d247-480a-82d4-56c3621a6b94-kube-api-access-4qlx4\") pod \"3c9ed414-d247-480a-82d4-56c3621a6b94\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") "
Nov 02 21:52:02.885529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.885423    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-openshift-ca\") pod \"3c9ed414-d247-480a-82d4-56c3621a6b94\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") "
Nov 02 21:52:02.885529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.885499    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-xks-ca\") pod \"3c9ed414-d247-480a-82d4-56c3621a6b94\" (UID: \"3c9ed414-d247-480a-82d4-56c3621a6b94\") "
Nov 02 21:52:02.885842 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.885819    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-openshift-ca" pod "3c9ed414-d247-480a-82d4-56c3621a6b94" (UID: "3c9ed414-d247-480a-82d4-56c3621a6b94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:52:02.886004 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.885991    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:02.887604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.887575    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-xks-ca" pod "3c9ed414-d247-480a-82d4-56c3621a6b94" (UID: "3c9ed414-d247-480a-82d4-56c3621a6b94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:52:02.887897 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.887834    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9ed414-d247-480a-82d4-56c3621a6b94-kube-api-access-4qlx4" pod "3c9ed414-d247-480a-82d4-56c3621a6b94" (UID: "3c9ed414-d247-480a-82d4-56c3621a6b94"). InnerVolumeSpecName "kube-api-access-4qlx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:52:03.012828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.990345    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qlx4\" (UniqueName: \"kubernetes.io/projected/3c9ed414-d247-480a-82d4-56c3621a6b94-kube-api-access-4qlx4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:03.012828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:02.990380    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ed414-d247-480a-82d4-56c3621a6b94-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:03.647791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:03.647759    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"
Nov 02 21:52:03.647791 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:03.647779    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg" event={"ID":"3c9ed414-d247-480a-82d4-56c3621a6b94","Type":"ContainerDied","Data":"f032622b5830de4afaf75e7263f073d5c933d608f70d72dbe08b0a9f83bf4b5f"}
Nov 02 21:52:03.648084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:03.647827    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f032622b5830de4afaf75e7263f073d5c933d608f70d72dbe08b0a9f83bf4b5f"
Nov 02 21:52:11.049135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:11.049091    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"]
Nov 02 21:52:11.099140 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:11.099098    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40723a0c-s6szl"]
Nov 02 21:52:12.952163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:12.946373    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c11505-b89a-4d53-aee8-b5ee3ad990f2" path="/var/lib/kubelet/pods/a5c11505-b89a-4d53-aee8-b5ee3ad990f2/volumes"
Nov 02 21:52:14.112713 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:52:14.112680    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:52:14.914828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:14.914799    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"]
Nov 02 21:52:14.915307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:14.915293    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3c9ed414-d247-480a-82d4-56c3621a6b94" containerName="revoke-keys"
Nov 02 21:52:14.915361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:14.915308    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9ed414-d247-480a-82d4-56c3621a6b94" containerName="revoke-keys"
Nov 02 21:52:14.915404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:14.915395    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9ed414-d247-480a-82d4-56c3621a6b94" containerName="revoke-keys"
Nov 02 21:52:14.920051 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:14.920036    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:14.970485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:14.970455    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"]
Nov 02 21:52:14.997532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:14.997500    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wx6d\" (UniqueName: \"kubernetes.io/projected/9b5b2434-443f-4a04-8275-5e1a78f967c7-kube-api-access-9wx6d\") pod \"maas-api-revoke-sub-d4ed5ce0-d2528\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:14.997692 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:14.997639    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ed5ce0-d2528\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:14.997764 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:14.997697    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ed5ce0-d2528\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:15.013420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.013397    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:52:15.013420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.013396    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:52:15.022539 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.022522    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:52:15.022597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.022531    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:52:15.098719 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.098698    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wx6d\" (UniqueName: \"kubernetes.io/projected/9b5b2434-443f-4a04-8275-5e1a78f967c7-kube-api-access-9wx6d\") pod \"maas-api-revoke-sub-d4ed5ce0-d2528\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:15.098834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.098746    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ed5ce0-d2528\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:15.098834 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.098784    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ed5ce0-d2528\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:15.099560 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.099538    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ed5ce0-d2528\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:15.101053 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.101036    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ed5ce0-d2528\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:15.155669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.155639    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wx6d\" (UniqueName: \"kubernetes.io/projected/9b5b2434-443f-4a04-8275-5e1a78f967c7-kube-api-access-9wx6d\") pod \"maas-api-revoke-sub-d4ed5ce0-d2528\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:15.233380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.233075    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:15.571977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.571944    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"]
Nov 02 21:52:15.577244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.577223    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.603592 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:15.603562    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5b2434_443f_4a04_8275_5e1a78f967c7.slice/crio-286c768b355594f95ad723be128170fbeb4ec121823e523fab42be5c1f30afc2 WatchSource:0}: Error finding container 286c768b355594f95ad723be128170fbeb4ec121823e523fab42be5c1f30afc2: Status 404 returned error can't find the container with id 286c768b355594f95ad723be128170fbeb4ec121823e523fab42be5c1f30afc2
Nov 02 21:52:15.606366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.606336    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"]
Nov 02 21:52:15.610369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.610345    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqs2v\" (UniqueName: \"kubernetes.io/projected/a20006e5-e52b-4756-a828-20cfbd1ad818-kube-api-access-fqs2v\") pod \"maas-api-revoke-sub-a3ad956c-29727\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.610540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.610520    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ad956c-29727\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.610616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.610558    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ad956c-29727\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.627752 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.627728    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"]
Nov 02 21:52:15.711792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.711752    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqs2v\" (UniqueName: \"kubernetes.io/projected/a20006e5-e52b-4756-a828-20cfbd1ad818-kube-api-access-fqs2v\") pod \"maas-api-revoke-sub-a3ad956c-29727\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.711963 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.711889    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ad956c-29727\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.711963 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.711914    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ad956c-29727\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.712648 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.712629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ad956c-29727\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.714060 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.714045    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ad956c-29727\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.745462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.745437    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528" event={"ID":"9b5b2434-443f-4a04-8275-5e1a78f967c7","Type":"ContainerStarted","Data":"286c768b355594f95ad723be128170fbeb4ec121823e523fab42be5c1f30afc2"}
Nov 02 21:52:15.769994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.769973    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqs2v\" (UniqueName: \"kubernetes.io/projected/a20006e5-e52b-4756-a828-20cfbd1ad818-kube-api-access-fqs2v\") pod \"maas-api-revoke-sub-a3ad956c-29727\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:15.890261 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:15.890197    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:16.233058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.232988    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"]
Nov 02 21:52:16.236992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.236968    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.262252 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:16.262226    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20006e5_e52b_4756_a828_20cfbd1ad818.slice/crio-37b4ded5764dddc55dbbd6636ddfffa8cd62c7954186501e575821f4af6e9588 WatchSource:0}: Error finding container 37b4ded5764dddc55dbbd6636ddfffa8cd62c7954186501e575821f4af6e9588: Status 404 returned error can't find the container with id 37b4ded5764dddc55dbbd6636ddfffa8cd62c7954186501e575821f4af6e9588
Nov 02 21:52:16.266819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.266765    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"]
Nov 02 21:52:16.293463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.293439    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"]
Nov 02 21:52:16.319571 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.319544    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.319705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.319588    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646ad9a4-208c-4d2b-9577-7599a6426125-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.319705 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.319684    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mtn\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-kube-api-access-j8mtn\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.319838 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.319783    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.319894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.319837    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.319954 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.319909    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646ad9a4-208c-4d2b-9577-7599a6426125-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.320068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.320048    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.320223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.320153    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.320223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.320179    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421219    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421254    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421301    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421325    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646ad9a4-208c-4d2b-9577-7599a6426125-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421354    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mtn\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-kube-api-access-j8mtn\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421402    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421428    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421445    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646ad9a4-208c-4d2b-9577-7599a6426125-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421468    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.421775 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.421677    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.422146 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.422098    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646ad9a4-208c-4d2b-9577-7599a6426125-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.422337 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.422320    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.422425 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.422402    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.422527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.422504    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.423713 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.423698    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646ad9a4-208c-4d2b-9577-7599a6426125-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.424057 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.424041    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.477555 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.477529    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.477651 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.477556    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mtn\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-kube-api-access-j8mtn\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.551522 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.551501    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:16.753304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.753222    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727" event={"ID":"a20006e5-e52b-4756-a828-20cfbd1ad818","Type":"ContainerStarted","Data":"aac6da1547cb026e2408bfbf2b1c9e374827b2a07ac7124d15bf0042c3ceccae"}
Nov 02 21:52:16.753304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.753273    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727" event={"ID":"a20006e5-e52b-4756-a828-20cfbd1ad818","Type":"ContainerStarted","Data":"37b4ded5764dddc55dbbd6636ddfffa8cd62c7954186501e575821f4af6e9588"}
Nov 02 21:52:16.754815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.754786    4528 generic.go:356] "Generic (PLEG): container finished" podID="9b5b2434-443f-4a04-8275-5e1a78f967c7" containerID="32d85dd25de2cc29420ae588ea74ea9f555ade29982e4ceb2cfaf4a9ee9774c6" exitCode=0
Nov 02 21:52:16.754943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.754837    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528" event={"ID":"9b5b2434-443f-4a04-8275-5e1a78f967c7","Type":"ContainerDied","Data":"32d85dd25de2cc29420ae588ea74ea9f555ade29982e4ceb2cfaf4a9ee9774c6"}
Nov 02 21:52:16.870913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.870858    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727" podStartSLOduration=1.870844282 podStartE2EDuration="1.870844282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:16.868983425 +0000 UTC m=+2624.132519742" watchObservedRunningTime="2026-11-02 21:52:16.870844282 +0000 UTC m=+2624.134380598"
Nov 02 21:52:16.967101 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:16.967073    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646ad9a4_208c_4d2b_9577_7599a6426125.slice/crio-c9082513d69d52461c75b8619c6b94a37ba6e0cafda21a98dbaaa98e9735df1d WatchSource:0}: Error finding container c9082513d69d52461c75b8619c6b94a37ba6e0cafda21a98dbaaa98e9735df1d: Status 404 returned error can't find the container with id c9082513d69d52461c75b8619c6b94a37ba6e0cafda21a98dbaaa98e9735df1d
Nov 02 21:52:16.969529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.969504    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"]
Nov 02 21:52:16.970042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.970010    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:52:16.970176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.970081    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:52:16.970176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:16.970144    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:52:17.763290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.763258    4528 generic.go:356] "Generic (PLEG): container finished" podID="a20006e5-e52b-4756-a828-20cfbd1ad818" containerID="aac6da1547cb026e2408bfbf2b1c9e374827b2a07ac7124d15bf0042c3ceccae" exitCode=0
Nov 02 21:52:17.763720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.763345    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727" event={"ID":"a20006e5-e52b-4756-a828-20cfbd1ad818","Type":"ContainerDied","Data":"aac6da1547cb026e2408bfbf2b1c9e374827b2a07ac7124d15bf0042c3ceccae"}
Nov 02 21:52:17.765080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.765059    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" event={"ID":"646ad9a4-208c-4d2b-9577-7599a6426125","Type":"ContainerStarted","Data":"78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771"}
Nov 02 21:52:17.765180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.765095    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" event={"ID":"646ad9a4-208c-4d2b-9577-7599a6426125","Type":"ContainerStarted","Data":"c9082513d69d52461c75b8619c6b94a37ba6e0cafda21a98dbaaa98e9735df1d"}
Nov 02 21:52:17.931128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.925357    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:17.939206 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.939177    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-openshift-ca\") pod \"9b5b2434-443f-4a04-8275-5e1a78f967c7\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") "
Nov 02 21:52:17.939376 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.939357    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-xks-ca\") pod \"9b5b2434-443f-4a04-8275-5e1a78f967c7\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") "
Nov 02 21:52:17.939454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.939436    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5b2434-443f-4a04-8275-5e1a78f967c7-kube-api-access-9wx6d\" (UniqueName: \"kubernetes.io/projected/9b5b2434-443f-4a04-8275-5e1a78f967c7-kube-api-access-9wx6d\") pod \"9b5b2434-443f-4a04-8275-5e1a78f967c7\" (UID: \"9b5b2434-443f-4a04-8275-5e1a78f967c7\") "
Nov 02 21:52:17.939573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.939553    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-openshift-ca" pod "9b5b2434-443f-4a04-8275-5e1a78f967c7" (UID: "9b5b2434-443f-4a04-8275-5e1a78f967c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:52:17.940006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.939979    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:17.941409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.941383    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5b2434-443f-4a04-8275-5e1a78f967c7-kube-api-access-9wx6d" pod "9b5b2434-443f-4a04-8275-5e1a78f967c7" (UID: "9b5b2434-443f-4a04-8275-5e1a78f967c7"). InnerVolumeSpecName "kube-api-access-9wx6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:52:17.941556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.941543    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-xks-ca" pod "9b5b2434-443f-4a04-8275-5e1a78f967c7" (UID: "9b5b2434-443f-4a04-8275-5e1a78f967c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:52:17.996973 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:17.996929    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" podStartSLOduration=1.996917631 podStartE2EDuration="1.996917631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:17.995468426 +0000 UTC m=+2625.259004756" watchObservedRunningTime="2026-11-02 21:52:17.996917631 +0000 UTC m=+2625.260453926"
Nov 02 21:52:18.041139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:18.041048    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5b2434-443f-4a04-8275-5e1a78f967c7-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:18.041139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:18.041076    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wx6d\" (UniqueName: \"kubernetes.io/projected/9b5b2434-443f-4a04-8275-5e1a78f967c7-kube-api-access-9wx6d\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:18.552302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:18.552261    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:18.554064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:18.554038    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 21:52:18.554216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:18.554085    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 21:52:18.771976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:18.771947    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"
Nov 02 21:52:18.772394 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:18.771981    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528" event={"ID":"9b5b2434-443f-4a04-8275-5e1a78f967c7","Type":"ContainerDied","Data":"286c768b355594f95ad723be128170fbeb4ec121823e523fab42be5c1f30afc2"}
Nov 02 21:52:18.772394 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:18.772029    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="286c768b355594f95ad723be128170fbeb4ec121823e523fab42be5c1f30afc2"
Nov 02 21:52:18.975410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:18.975383    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:19.051936 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.051907    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a20006e5-e52b-4756-a828-20cfbd1ad818-kube-api-access-fqs2v\" (UniqueName: \"kubernetes.io/projected/a20006e5-e52b-4756-a828-20cfbd1ad818-kube-api-access-fqs2v\") pod \"a20006e5-e52b-4756-a828-20cfbd1ad818\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") "
Nov 02 21:52:19.052127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.052073    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-xks-ca\") pod \"a20006e5-e52b-4756-a828-20cfbd1ad818\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") "
Nov 02 21:52:19.052127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.052100    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-openshift-ca\") pod \"a20006e5-e52b-4756-a828-20cfbd1ad818\" (UID: \"a20006e5-e52b-4756-a828-20cfbd1ad818\") "
Nov 02 21:52:19.052435 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.052402    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-openshift-ca" pod "a20006e5-e52b-4756-a828-20cfbd1ad818" (UID: "a20006e5-e52b-4756-a828-20cfbd1ad818"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:52:19.052631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.052614    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:19.053997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.053977    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-xks-ca" pod "a20006e5-e52b-4756-a828-20cfbd1ad818" (UID: "a20006e5-e52b-4756-a828-20cfbd1ad818"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:52:19.054079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.054045    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20006e5-e52b-4756-a828-20cfbd1ad818-kube-api-access-fqs2v" pod "a20006e5-e52b-4756-a828-20cfbd1ad818" (UID: "a20006e5-e52b-4756-a828-20cfbd1ad818"). InnerVolumeSpecName "kube-api-access-fqs2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:52:19.153924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.153844    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20006e5-e52b-4756-a828-20cfbd1ad818-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:19.153924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.153870    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqs2v\" (UniqueName: \"kubernetes.io/projected/a20006e5-e52b-4756-a828-20cfbd1ad818-kube-api-access-fqs2v\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:19.552841 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.552808    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 21:52:19.553027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.552858    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 21:52:19.779134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.779079    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"
Nov 02 21:52:19.779134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.779098    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727" event={"ID":"a20006e5-e52b-4756-a828-20cfbd1ad818","Type":"ContainerDied","Data":"37b4ded5764dddc55dbbd6636ddfffa8cd62c7954186501e575821f4af6e9588"}
Nov 02 21:52:19.779587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:19.779169    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37b4ded5764dddc55dbbd6636ddfffa8cd62c7954186501e575821f4af6e9588"
Nov 02 21:52:20.551843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.551814    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 21:52:20.551998 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.551862    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 21:52:20.690919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.690888    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"]
Nov 02 21:52:20.691460 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.691447    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a20006e5-e52b-4756-a828-20cfbd1ad818" containerName="revoke-keys"
Nov 02 21:52:20.691514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.691462    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a20006e5-e52b-4756-a828-20cfbd1ad818" containerName="revoke-keys"
Nov 02 21:52:20.691514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.691479    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9b5b2434-443f-4a04-8275-5e1a78f967c7" containerName="revoke-keys"
Nov 02 21:52:20.691514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.691484    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5b2434-443f-4a04-8275-5e1a78f967c7" containerName="revoke-keys"
Nov 02 21:52:20.691615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.691566    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a20006e5-e52b-4756-a828-20cfbd1ad818" containerName="revoke-keys"
Nov 02 21:52:20.691615 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.691577    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5b2434-443f-4a04-8275-5e1a78f967c7" containerName="revoke-keys"
Nov 02 21:52:20.695195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.695173    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:20.746225 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.746196    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"]
Nov 02 21:52:20.771409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.771371    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df8kz\" (UniqueName: \"kubernetes.io/projected/a40bb762-0d41-490d-baac-b963dfd30fd5-kube-api-access-df8kz\") pod \"maas-api-revoke-sub-225d1433-vq8rz\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:20.771602 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.771582    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225d1433-vq8rz\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:20.771671 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.771656    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225d1433-vq8rz\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:20.792639 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.792611    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:52:20.792639 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.792612    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:52:20.792639 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.792612    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:52:20.793064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.792652    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:52:20.872978 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.872887    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225d1433-vq8rz\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:20.872978 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.872951    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df8kz\" (UniqueName: \"kubernetes.io/projected/a40bb762-0d41-490d-baac-b963dfd30fd5-kube-api-access-df8kz\") pod \"maas-api-revoke-sub-225d1433-vq8rz\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:20.873269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.873072    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225d1433-vq8rz\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:20.873657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.873636    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225d1433-vq8rz\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:20.875211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.875189    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225d1433-vq8rz\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:20.944482 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:20.937092    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df8kz\" (UniqueName: \"kubernetes.io/projected/a40bb762-0d41-490d-baac-b963dfd30fd5-kube-api-access-df8kz\") pod \"maas-api-revoke-sub-225d1433-vq8rz\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:21.029431 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:21.029398    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:21.400960 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:21.400923    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40bb762_0d41_490d_baac_b963dfd30fd5.slice/crio-608444146a55decd0934429a52f9f5bc0e546b0a39d53be9f65ef53075eb92f3 WatchSource:0}: Error finding container 608444146a55decd0934429a52f9f5bc0e546b0a39d53be9f65ef53075eb92f3: Status 404 returned error can't find the container with id 608444146a55decd0934429a52f9f5bc0e546b0a39d53be9f65ef53075eb92f3
Nov 02 21:52:21.433129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:21.431279    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"]
Nov 02 21:52:21.552582 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:21.552553    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 21:52:21.552742 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:21.552598    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 21:52:21.793102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:21.793065    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz" event={"ID":"a40bb762-0d41-490d-baac-b963dfd30fd5","Type":"ContainerStarted","Data":"608444146a55decd0934429a52f9f5bc0e546b0a39d53be9f65ef53075eb92f3"}
Nov 02 21:52:22.552237 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:22.552205    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 21:52:22.552404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:22.552258    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 21:52:22.800456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:22.800420    4528 generic.go:356] "Generic (PLEG): container finished" podID="a40bb762-0d41-490d-baac-b963dfd30fd5" containerID="96bc46c764375b64e39953e77bd2577ba339d03a36fbeb99aa409174f4dc1dee" exitCode=0
Nov 02 21:52:22.800852 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:22.800474    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz" event={"ID":"a40bb762-0d41-490d-baac-b963dfd30fd5","Type":"ContainerDied","Data":"96bc46c764375b64e39953e77bd2577ba339d03a36fbeb99aa409174f4dc1dee"}
Nov 02 21:52:23.552009 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:23.551970    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 21:52:23.552211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:23.552019    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 21:52:23.961404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:23.961380    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:24.033538 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.033504    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-openshift-ca\") pod \"a40bb762-0d41-490d-baac-b963dfd30fd5\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") "
Nov 02 21:52:24.033722 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.033639    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-xks-ca\") pod \"a40bb762-0d41-490d-baac-b963dfd30fd5\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") "
Nov 02 21:52:24.033821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.033762    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40bb762-0d41-490d-baac-b963dfd30fd5-kube-api-access-df8kz\" (UniqueName: \"kubernetes.io/projected/a40bb762-0d41-490d-baac-b963dfd30fd5-kube-api-access-df8kz\") pod \"a40bb762-0d41-490d-baac-b963dfd30fd5\" (UID: \"a40bb762-0d41-490d-baac-b963dfd30fd5\") "
Nov 02 21:52:24.033966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.033936    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-openshift-ca" pod "a40bb762-0d41-490d-baac-b963dfd30fd5" (UID: "a40bb762-0d41-490d-baac-b963dfd30fd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:52:24.034694 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.034673    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:24.035987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.035961    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-xks-ca" pod "a40bb762-0d41-490d-baac-b963dfd30fd5" (UID: "a40bb762-0d41-490d-baac-b963dfd30fd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:52:24.036454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.036429    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40bb762-0d41-490d-baac-b963dfd30fd5-kube-api-access-df8kz" pod "a40bb762-0d41-490d-baac-b963dfd30fd5" (UID: "a40bb762-0d41-490d-baac-b963dfd30fd5"). InnerVolumeSpecName "kube-api-access-df8kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:52:24.136156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.136067    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40bb762-0d41-490d-baac-b963dfd30fd5-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:24.136156 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.136100    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df8kz\" (UniqueName: \"kubernetes.io/projected/a40bb762-0d41-490d-baac-b963dfd30fd5-kube-api-access-df8kz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:24.556447 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.556419    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:24.556926 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.556901    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:24.557644 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.557624    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 21:52:24.815338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.815259    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"
Nov 02 21:52:24.815338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.815283    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz" event={"ID":"a40bb762-0d41-490d-baac-b963dfd30fd5","Type":"ContainerDied","Data":"608444146a55decd0934429a52f9f5bc0e546b0a39d53be9f65ef53075eb92f3"}
Nov 02 21:52:24.815338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:24.815329    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="608444146a55decd0934429a52f9f5bc0e546b0a39d53be9f65ef53075eb92f3"
Nov 02 21:52:26.362299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.362269    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"]
Nov 02 21:52:26.362778 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.362765    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a40bb762-0d41-490d-baac-b963dfd30fd5" containerName="revoke-keys"
Nov 02 21:52:26.362829 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.362779    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40bb762-0d41-490d-baac-b963dfd30fd5" containerName="revoke-keys"
Nov 02 21:52:26.362894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.362885    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40bb762-0d41-490d-baac-b963dfd30fd5" containerName="revoke-keys"
Nov 02 21:52:26.366365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.366346    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.418984 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.418946    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"]
Nov 02 21:52:26.465200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.465174    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:52:26.465200 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.465197    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:52:26.465399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.465218    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:52:26.468723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.468707    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:52:26.496992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.496962    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b8802b5-mjc9c\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.497133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.497028    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b8802b5-mjc9c\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.497187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.497157    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wlr5\" (UniqueName: \"kubernetes.io/projected/d87471c0-823c-454f-8bbd-e976b1f7d3fe-kube-api-access-4wlr5\") pod \"maas-api-revoke-sub-6b8802b5-mjc9c\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.598009 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.597981    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b8802b5-mjc9c\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.598178 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.598031    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b8802b5-mjc9c\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.598178 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.598074    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wlr5\" (UniqueName: \"kubernetes.io/projected/d87471c0-823c-454f-8bbd-e976b1f7d3fe-kube-api-access-4wlr5\") pod \"maas-api-revoke-sub-6b8802b5-mjc9c\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.598614 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.598585    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b8802b5-mjc9c\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.600273 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.600252    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b8802b5-mjc9c\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.654667 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.654605    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wlr5\" (UniqueName: \"kubernetes.io/projected/d87471c0-823c-454f-8bbd-e976b1f7d3fe-kube-api-access-4wlr5\") pod \"maas-api-revoke-sub-6b8802b5-mjc9c\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:26.679455 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:26.679436    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:27.048963 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:27.048934    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87471c0_823c_454f_8bbd_e976b1f7d3fe.slice/crio-bfdcb3d6313dcf104440811e33c7ee795c0b890f58a0db87b4d199accdcd8a36 WatchSource:0}: Error finding container bfdcb3d6313dcf104440811e33c7ee795c0b890f58a0db87b4d199accdcd8a36: Status 404 returned error can't find the container with id bfdcb3d6313dcf104440811e33c7ee795c0b890f58a0db87b4d199accdcd8a36
Nov 02 21:52:27.052027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:27.052003    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"]
Nov 02 21:52:27.874232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:27.874204    4528 generic.go:356] "Generic (PLEG): container finished" podID="d87471c0-823c-454f-8bbd-e976b1f7d3fe" containerID="1911cb92a445fdebf696c3c27c0d6c088a46eea5cef9306f4ade5604cf2366b0" exitCode=0
Nov 02 21:52:27.874547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:27.874282    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c" event={"ID":"d87471c0-823c-454f-8bbd-e976b1f7d3fe","Type":"ContainerDied","Data":"1911cb92a445fdebf696c3c27c0d6c088a46eea5cef9306f4ade5604cf2366b0"}
Nov 02 21:52:27.874547 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:27.874336    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c" event={"ID":"d87471c0-823c-454f-8bbd-e976b1f7d3fe","Type":"ContainerStarted","Data":"bfdcb3d6313dcf104440811e33c7ee795c0b890f58a0db87b4d199accdcd8a36"}
Nov 02 21:52:29.027400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.027377    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:29.131870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.131836    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-xks-ca\") pod \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") "
Nov 02 21:52:29.132050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.131917    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-openshift-ca\") pod \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") "
Nov 02 21:52:29.132050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.131980    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87471c0-823c-454f-8bbd-e976b1f7d3fe-kube-api-access-4wlr5\" (UniqueName: \"kubernetes.io/projected/d87471c0-823c-454f-8bbd-e976b1f7d3fe-kube-api-access-4wlr5\") pod \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\" (UID: \"d87471c0-823c-454f-8bbd-e976b1f7d3fe\") "
Nov 02 21:52:29.132379 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.132344    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-openshift-ca" pod "d87471c0-823c-454f-8bbd-e976b1f7d3fe" (UID: "d87471c0-823c-454f-8bbd-e976b1f7d3fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:52:29.132639 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.132592    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:29.134007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.133988    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87471c0-823c-454f-8bbd-e976b1f7d3fe-kube-api-access-4wlr5" pod "d87471c0-823c-454f-8bbd-e976b1f7d3fe" (UID: "d87471c0-823c-454f-8bbd-e976b1f7d3fe"). InnerVolumeSpecName "kube-api-access-4wlr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:52:29.134068 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.133997    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-xks-ca" pod "d87471c0-823c-454f-8bbd-e976b1f7d3fe" (UID: "d87471c0-823c-454f-8bbd-e976b1f7d3fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:52:29.233511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.233481    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wlr5\" (UniqueName: \"kubernetes.io/projected/d87471c0-823c-454f-8bbd-e976b1f7d3fe-kube-api-access-4wlr5\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:29.233511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.233506    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d87471c0-823c-454f-8bbd-e976b1f7d3fe-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:29.889743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.889713    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"
Nov 02 21:52:29.889922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.889744    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c" event={"ID":"d87471c0-823c-454f-8bbd-e976b1f7d3fe","Type":"ContainerDied","Data":"bfdcb3d6313dcf104440811e33c7ee795c0b890f58a0db87b4d199accdcd8a36"}
Nov 02 21:52:29.889922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:29.889813    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfdcb3d6313dcf104440811e33c7ee795c0b890f58a0db87b4d199accdcd8a36"
Nov 02 21:52:30.349278 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.349244    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"]
Nov 02 21:52:30.349796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.349777    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d87471c0-823c-454f-8bbd-e976b1f7d3fe" containerName="revoke-keys"
Nov 02 21:52:30.349862 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.349797    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87471c0-823c-454f-8bbd-e976b1f7d3fe" containerName="revoke-keys"
Nov 02 21:52:30.349917 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.349904    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87471c0-823c-454f-8bbd-e976b1f7d3fe" containerName="revoke-keys"
Nov 02 21:52:30.353378 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.353359    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.405899 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.405868    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:52:30.406179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.406157    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:52:30.407231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.407203    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"]
Nov 02 21:52:30.451928 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.451907    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:52:30.452033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.451993    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:52:30.457720 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.457701    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03be370e-v252m\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.457781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.457752    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03be370e-v252m\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.457861 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.457841    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggv6t\" (UniqueName: \"kubernetes.io/projected/f3423c69-3870-44e7-aee8-8d30cfe87bf6-kube-api-access-ggv6t\") pod \"maas-api-revoke-sub-03be370e-v252m\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.559397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.559370    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03be370e-v252m\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.559526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.559418    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03be370e-v252m\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.559526 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.559443    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggv6t\" (UniqueName: \"kubernetes.io/projected/f3423c69-3870-44e7-aee8-8d30cfe87bf6-kube-api-access-ggv6t\") pod \"maas-api-revoke-sub-03be370e-v252m\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.560086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.560064    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03be370e-v252m\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.561496 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.561475    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03be370e-v252m\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.616163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.616093    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggv6t\" (UniqueName: \"kubernetes.io/projected/f3423c69-3870-44e7-aee8-8d30cfe87bf6-kube-api-access-ggv6t\") pod \"maas-api-revoke-sub-03be370e-v252m\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:30.667036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:30.667009    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:31.035829 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:31.035800    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3423c69_3870_44e7_aee8_8d30cfe87bf6.slice/crio-270fc553a5c934df12e2284f3cc7fad02a7880dc9bedd353f7f9592ea135d9ab WatchSource:0}: Error finding container 270fc553a5c934df12e2284f3cc7fad02a7880dc9bedd353f7f9592ea135d9ab: Status 404 returned error can't find the container with id 270fc553a5c934df12e2284f3cc7fad02a7880dc9bedd353f7f9592ea135d9ab
Nov 02 21:52:31.037412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.037393    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"]
Nov 02 21:52:31.903619 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.903589    4528 generic.go:356] "Generic (PLEG): container finished" podID="f3423c69-3870-44e7-aee8-8d30cfe87bf6" containerID="2eef7cf059f9be9e2f1f2be6961e657c82c7c22220021db8f7b4150ef288552d" exitCode=0
Nov 02 21:52:31.903962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.903669    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m" event={"ID":"f3423c69-3870-44e7-aee8-8d30cfe87bf6","Type":"ContainerDied","Data":"2eef7cf059f9be9e2f1f2be6961e657c82c7c22220021db8f7b4150ef288552d"}
Nov 02 21:52:31.903962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.903753    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m" event={"ID":"f3423c69-3870-44e7-aee8-8d30cfe87bf6","Type":"ContainerStarted","Data":"270fc553a5c934df12e2284f3cc7fad02a7880dc9bedd353f7f9592ea135d9ab"}
Nov 02 21:52:31.911594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.911576    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"]
Nov 02 21:52:31.915498 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.915484    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:31.968390 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.968364    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"]
Nov 02 21:52:31.975420 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.975392    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab41bd83-ktr4r\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:31.975587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.975560    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab41bd83-ktr4r\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:31.975820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:31.975797    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rp2\" (UniqueName: \"kubernetes.io/projected/c10c5b99-4438-4952-99aa-6a49a69f5de4-kube-api-access-75rp2\") pod \"maas-api-revoke-sub-ab41bd83-ktr4r\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:32.077391 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:32.077359    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rp2\" (UniqueName: \"kubernetes.io/projected/c10c5b99-4438-4952-99aa-6a49a69f5de4-kube-api-access-75rp2\") pod \"maas-api-revoke-sub-ab41bd83-ktr4r\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:32.077532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:32.077475    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab41bd83-ktr4r\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:32.077532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:32.077494    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab41bd83-ktr4r\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:32.078053 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:32.078035    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab41bd83-ktr4r\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:32.079754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:32.079736    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab41bd83-ktr4r\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:32.137380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:32.137312    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rp2\" (UniqueName: \"kubernetes.io/projected/c10c5b99-4438-4952-99aa-6a49a69f5de4-kube-api-access-75rp2\") pod \"maas-api-revoke-sub-ab41bd83-ktr4r\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:32.263259 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:32.263234    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:32.629906 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:32.629875    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10c5b99_4438_4952_99aa_6a49a69f5de4.slice/crio-4ccb90cb95a24694a6a0eb1fe469be8194ec0989b1f1c4aa6d0b7e7a3e6b577d WatchSource:0}: Error finding container 4ccb90cb95a24694a6a0eb1fe469be8194ec0989b1f1c4aa6d0b7e7a3e6b577d: Status 404 returned error can't find the container with id 4ccb90cb95a24694a6a0eb1fe469be8194ec0989b1f1c4aa6d0b7e7a3e6b577d
Nov 02 21:52:32.633086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:32.633062    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"]
Nov 02 21:52:32.914678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:32.914648    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r" event={"ID":"c10c5b99-4438-4952-99aa-6a49a69f5de4","Type":"ContainerStarted","Data":"4ccb90cb95a24694a6a0eb1fe469be8194ec0989b1f1c4aa6d0b7e7a3e6b577d"}
Nov 02 21:52:33.057751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.057692    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:33.194440 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.194361    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-openshift-ca\") pod \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") "
Nov 02 21:52:33.194607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.194515    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3423c69-3870-44e7-aee8-8d30cfe87bf6-kube-api-access-ggv6t\" (UniqueName: \"kubernetes.io/projected/f3423c69-3870-44e7-aee8-8d30cfe87bf6-kube-api-access-ggv6t\") pod \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") "
Nov 02 21:52:33.194607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.194567    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-xks-ca\") pod \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\" (UID: \"f3423c69-3870-44e7-aee8-8d30cfe87bf6\") "
Nov 02 21:52:33.194767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.194739    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-openshift-ca" pod "f3423c69-3870-44e7-aee8-8d30cfe87bf6" (UID: "f3423c69-3870-44e7-aee8-8d30cfe87bf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:52:33.195048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.195022    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:33.196656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.196626    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-xks-ca" pod "f3423c69-3870-44e7-aee8-8d30cfe87bf6" (UID: "f3423c69-3870-44e7-aee8-8d30cfe87bf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:52:33.196757 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.196655    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3423c69-3870-44e7-aee8-8d30cfe87bf6-kube-api-access-ggv6t" pod "f3423c69-3870-44e7-aee8-8d30cfe87bf6" (UID: "f3423c69-3870-44e7-aee8-8d30cfe87bf6"). InnerVolumeSpecName "kube-api-access-ggv6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:52:33.295924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.295897    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggv6t\" (UniqueName: \"kubernetes.io/projected/f3423c69-3870-44e7-aee8-8d30cfe87bf6-kube-api-access-ggv6t\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:33.295924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.295924    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3423c69-3870-44e7-aee8-8d30cfe87bf6-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:33.922411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.922378    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"
Nov 02 21:52:33.922411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.922397    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m" event={"ID":"f3423c69-3870-44e7-aee8-8d30cfe87bf6","Type":"ContainerDied","Data":"270fc553a5c934df12e2284f3cc7fad02a7880dc9bedd353f7f9592ea135d9ab"}
Nov 02 21:52:33.922970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.922453    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="270fc553a5c934df12e2284f3cc7fad02a7880dc9bedd353f7f9592ea135d9ab"
Nov 02 21:52:33.924313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.924283    4528 generic.go:356] "Generic (PLEG): container finished" podID="c10c5b99-4438-4952-99aa-6a49a69f5de4" containerID="e1f55efac5b939d28a5ef3558b79f8c67d0dbfa69cd52f35a56185fb138f646a" exitCode=0
Nov 02 21:52:33.924456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:33.924348    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r" event={"ID":"c10c5b99-4438-4952-99aa-6a49a69f5de4","Type":"ContainerDied","Data":"e1f55efac5b939d28a5ef3558b79f8c67d0dbfa69cd52f35a56185fb138f646a"}
Nov 02 21:52:35.073361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.073337    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:35.116871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.116843    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10c5b99-4438-4952-99aa-6a49a69f5de4-kube-api-access-75rp2\" (UniqueName: \"kubernetes.io/projected/c10c5b99-4438-4952-99aa-6a49a69f5de4-kube-api-access-75rp2\") pod \"c10c5b99-4438-4952-99aa-6a49a69f5de4\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") "
Nov 02 21:52:35.117011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.116893    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-openshift-ca\") pod \"c10c5b99-4438-4952-99aa-6a49a69f5de4\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") "
Nov 02 21:52:35.117011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.116981    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-xks-ca\") pod \"c10c5b99-4438-4952-99aa-6a49a69f5de4\" (UID: \"c10c5b99-4438-4952-99aa-6a49a69f5de4\") "
Nov 02 21:52:35.117310 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.117284    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-openshift-ca" pod "c10c5b99-4438-4952-99aa-6a49a69f5de4" (UID: "c10c5b99-4438-4952-99aa-6a49a69f5de4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:52:35.117474 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.117457    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:35.118856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.118836    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10c5b99-4438-4952-99aa-6a49a69f5de4-kube-api-access-75rp2" pod "c10c5b99-4438-4952-99aa-6a49a69f5de4" (UID: "c10c5b99-4438-4952-99aa-6a49a69f5de4"). InnerVolumeSpecName "kube-api-access-75rp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:52:35.118987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.118971    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-xks-ca" pod "c10c5b99-4438-4952-99aa-6a49a69f5de4" (UID: "c10c5b99-4438-4952-99aa-6a49a69f5de4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:52:35.219160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.219066    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10c5b99-4438-4952-99aa-6a49a69f5de4-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:35.219160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.219091    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rp2\" (UniqueName: \"kubernetes.io/projected/c10c5b99-4438-4952-99aa-6a49a69f5de4-kube-api-access-75rp2\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:35.938624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.938585    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r" event={"ID":"c10c5b99-4438-4952-99aa-6a49a69f5de4","Type":"ContainerDied","Data":"4ccb90cb95a24694a6a0eb1fe469be8194ec0989b1f1c4aa6d0b7e7a3e6b577d"}
Nov 02 21:52:35.938807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.938639    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ccb90cb95a24694a6a0eb1fe469be8194ec0989b1f1c4aa6d0b7e7a3e6b577d"
Nov 02 21:52:35.938807 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:35.938598    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"
Nov 02 21:52:38.773333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:38.773294    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:52:41.407275 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:41.407215    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:52:44.133436 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:52:44.133390    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:52:45.961983 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:45.961945    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"]
Nov 02 21:52:45.962700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:45.962678    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f3423c69-3870-44e7-aee8-8d30cfe87bf6" containerName="revoke-keys"
Nov 02 21:52:45.962802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:45.962702    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3423c69-3870-44e7-aee8-8d30cfe87bf6" containerName="revoke-keys"
Nov 02 21:52:45.962802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:45.962727    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c10c5b99-4438-4952-99aa-6a49a69f5de4" containerName="revoke-keys"
Nov 02 21:52:45.962802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:45.962736    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10c5b99-4438-4952-99aa-6a49a69f5de4" containerName="revoke-keys"
Nov 02 21:52:45.962969 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:45.962860    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10c5b99-4438-4952-99aa-6a49a69f5de4" containerName="revoke-keys"
Nov 02 21:52:45.962969 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:45.962888    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3423c69-3870-44e7-aee8-8d30cfe87bf6" containerName="revoke-keys"
Nov 02 21:52:45.966820 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:45.966799    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.019604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.019573    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"]
Nov 02 21:52:46.065543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.065489    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:52:46.065543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.065532    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:52:46.065543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.065539    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:52:46.065821 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.065539    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:52:46.146534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.146506    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a873435-8cmpq\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.146706 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.146575    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a873435-8cmpq\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.146781 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.146696    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k6dm\" (UniqueName: \"kubernetes.io/projected/9c91b35c-d00f-4dbb-a601-b0f865238201-kube-api-access-6k6dm\") pod \"maas-api-revoke-sub-2a873435-8cmpq\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.247418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.247342    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a873435-8cmpq\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.247418 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.247378    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6dm\" (UniqueName: \"kubernetes.io/projected/9c91b35c-d00f-4dbb-a601-b0f865238201-kube-api-access-6k6dm\") pod \"maas-api-revoke-sub-2a873435-8cmpq\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.247641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.247490    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a873435-8cmpq\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.247996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.247972    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a873435-8cmpq\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.249611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.249588    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a873435-8cmpq\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.304594 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.304564    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6dm\" (UniqueName: \"kubernetes.io/projected/9c91b35c-d00f-4dbb-a601-b0f865238201-kube-api-access-6k6dm\") pod \"maas-api-revoke-sub-2a873435-8cmpq\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.580404 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.580372    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:46.966230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:46.966204    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"]
Nov 02 21:52:47.074457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:47.074415    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq" event={"ID":"9c91b35c-d00f-4dbb-a601-b0f865238201","Type":"ContainerStarted","Data":"c3e246f2507f386dda3f349ba87b6be5c7d6ce65a9f0064881aaf8f53906f11e"}
Nov 02 21:52:48.106131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:48.102962    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq" event={"ID":"9c91b35c-d00f-4dbb-a601-b0f865238201","Type":"ContainerStarted","Data":"56c59d8ea38754a80aef34341016286c808221bc5800b316b73b2e3eb74f586c"}
Nov 02 21:52:48.220942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:48.220879    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq" podStartSLOduration=3.220862584 podStartE2EDuration="3.220862584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:48.220600389 +0000 UTC m=+2655.484136706" watchObservedRunningTime="2026-11-02 21:52:48.220862584 +0000 UTC m=+2655.484398902"
Nov 02 21:52:48.936612 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:48.936578    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"]
Nov 02 21:52:48.940597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:48.940580    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:48.993042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:48.993007    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"]
Nov 02 21:52:49.022044 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.022011    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9b6a9c-8ftkq\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.022247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.022186    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9b6a9c-8ftkq\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.022542 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.022502    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvfn6\" (UniqueName: \"kubernetes.io/projected/5a65a47e-f350-4381-90d6-7aab121bd723-kube-api-access-tvfn6\") pod \"maas-api-revoke-sub-6e9b6a9c-8ftkq\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.111746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.111707    4528 generic.go:356] "Generic (PLEG): container finished" podID="9c91b35c-d00f-4dbb-a601-b0f865238201" containerID="56c59d8ea38754a80aef34341016286c808221bc5800b316b73b2e3eb74f586c" exitCode=0
Nov 02 21:52:49.112177 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.111784    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq" event={"ID":"9c91b35c-d00f-4dbb-a601-b0f865238201","Type":"ContainerDied","Data":"56c59d8ea38754a80aef34341016286c808221bc5800b316b73b2e3eb74f586c"}
Nov 02 21:52:49.124356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.124332    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9b6a9c-8ftkq\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.124533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.124512    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9b6a9c-8ftkq\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.124685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.124666    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvfn6\" (UniqueName: \"kubernetes.io/projected/5a65a47e-f350-4381-90d6-7aab121bd723-kube-api-access-tvfn6\") pod \"maas-api-revoke-sub-6e9b6a9c-8ftkq\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.124962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.124943    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9b6a9c-8ftkq\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.126848 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.126828    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9b6a9c-8ftkq\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.181143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.181105    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvfn6\" (UniqueName: \"kubernetes.io/projected/5a65a47e-f350-4381-90d6-7aab121bd723-kube-api-access-tvfn6\") pod \"maas-api-revoke-sub-6e9b6a9c-8ftkq\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.261595 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.254772    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:49.396288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.396257    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:52:49.398517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.396310    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:52:49.639892 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:49.639862    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a65a47e_f350_4381_90d6_7aab121bd723.slice/crio-839790f2b6ab67cd4f0db25bca77d7df2c4456e65749e1026efea7259dbf3be5 WatchSource:0}: Error finding container 839790f2b6ab67cd4f0db25bca77d7df2c4456e65749e1026efea7259dbf3be5: Status 404 returned error can't find the container with id 839790f2b6ab67cd4f0db25bca77d7df2c4456e65749e1026efea7259dbf3be5
Nov 02 21:52:49.643824 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:49.643801    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"]
Nov 02 21:52:50.120866 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.120834    4528 generic.go:356] "Generic (PLEG): container finished" podID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerID="538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c" exitCode=0
Nov 02 21:52:50.121393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.120859    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" event={"ID":"c8a95916-ee75-44bc-be9e-8cc8baed5bf1","Type":"ContainerDied","Data":"538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c"}
Nov 02 21:52:50.121393 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.121348    4528 scope.go:122] "RemoveContainer" containerID="538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c"
Nov 02 21:52:50.122989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.122890    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:52:50.122989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.122968    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:52:50.123210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.123011    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:52:50.123210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.123146    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq" event={"ID":"5a65a47e-f350-4381-90d6-7aab121bd723","Type":"ContainerStarted","Data":"701028591c62e641e608bc90bd4c60f78e8f005fbf4e32fe7fde03736f13e776"}
Nov 02 21:52:50.123210 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.123189    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq" event={"ID":"5a65a47e-f350-4381-90d6-7aab121bd723","Type":"ContainerStarted","Data":"839790f2b6ab67cd4f0db25bca77d7df2c4456e65749e1026efea7259dbf3be5"}
Nov 02 21:52:50.298588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.298566    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:50.363987 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.363926    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq" podStartSLOduration=2.3639125979999998 podStartE2EDuration="2.363912598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:50.361495394 +0000 UTC m=+2657.625031710" watchObservedRunningTime="2026-11-02 21:52:50.363912598 +0000 UTC m=+2657.627448915"
Nov 02 21:52:50.385438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.385410    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-xks-ca\") pod \"9c91b35c-d00f-4dbb-a601-b0f865238201\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") "
Nov 02 21:52:50.385601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.385532    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c91b35c-d00f-4dbb-a601-b0f865238201-kube-api-access-6k6dm\" (UniqueName: \"kubernetes.io/projected/9c91b35c-d00f-4dbb-a601-b0f865238201-kube-api-access-6k6dm\") pod \"9c91b35c-d00f-4dbb-a601-b0f865238201\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") "
Nov 02 21:52:50.385744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.385727    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-openshift-ca\") pod \"9c91b35c-d00f-4dbb-a601-b0f865238201\" (UID: \"9c91b35c-d00f-4dbb-a601-b0f865238201\") "
Nov 02 21:52:50.386142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.386088    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-openshift-ca" pod "9c91b35c-d00f-4dbb-a601-b0f865238201" (UID: "9c91b35c-d00f-4dbb-a601-b0f865238201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:52:50.387746 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.387715    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-xks-ca" pod "9c91b35c-d00f-4dbb-a601-b0f865238201" (UID: "9c91b35c-d00f-4dbb-a601-b0f865238201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:52:50.387867 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.387770    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c91b35c-d00f-4dbb-a601-b0f865238201-kube-api-access-6k6dm" pod "9c91b35c-d00f-4dbb-a601-b0f865238201" (UID: "9c91b35c-d00f-4dbb-a601-b0f865238201"). InnerVolumeSpecName "kube-api-access-6k6dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:52:50.487457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.487382    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:50.487457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.487419    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c91b35c-d00f-4dbb-a601-b0f865238201-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:50.487457 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:50.487436    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k6dm\" (UniqueName: \"kubernetes.io/projected/9c91b35c-d00f-4dbb-a601-b0f865238201-kube-api-access-6k6dm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:51.135024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:51.134988    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" event={"ID":"c8a95916-ee75-44bc-be9e-8cc8baed5bf1","Type":"ContainerStarted","Data":"5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c"}
Nov 02 21:52:51.136605 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:51.136577    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq" event={"ID":"9c91b35c-d00f-4dbb-a601-b0f865238201","Type":"ContainerDied","Data":"c3e246f2507f386dda3f349ba87b6be5c7d6ce65a9f0064881aaf8f53906f11e"}
Nov 02 21:52:51.136733 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:51.136634    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e246f2507f386dda3f349ba87b6be5c7d6ce65a9f0064881aaf8f53906f11e"
Nov 02 21:52:51.136733 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:51.136586    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"
Nov 02 21:52:51.138176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:51.138155    4528 generic.go:356] "Generic (PLEG): container finished" podID="5a65a47e-f350-4381-90d6-7aab121bd723" containerID="701028591c62e641e608bc90bd4c60f78e8f005fbf4e32fe7fde03736f13e776" exitCode=0
Nov 02 21:52:51.138283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:51.138240    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq" event={"ID":"5a65a47e-f350-4381-90d6-7aab121bd723","Type":"ContainerDied","Data":"701028591c62e641e608bc90bd4c60f78e8f005fbf4e32fe7fde03736f13e776"}
Nov 02 21:52:51.426132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:51.423677    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:52:51.426132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:51.425939    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 02 21:52:51.426132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:51.425988    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 02 21:52:52.348395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.348367    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:52.424306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.424227    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 02 21:52:52.424557 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.424530    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 02 21:52:52.436732 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.436710    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-openshift-ca\") pod \"5a65a47e-f350-4381-90d6-7aab121bd723\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") "
Nov 02 21:52:52.436863 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.436776    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a65a47e-f350-4381-90d6-7aab121bd723-kube-api-access-tvfn6\" (UniqueName: \"kubernetes.io/projected/5a65a47e-f350-4381-90d6-7aab121bd723-kube-api-access-tvfn6\") pod \"5a65a47e-f350-4381-90d6-7aab121bd723\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") "
Nov 02 21:52:52.436863 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.436819    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-xks-ca\") pod \"5a65a47e-f350-4381-90d6-7aab121bd723\" (UID: \"5a65a47e-f350-4381-90d6-7aab121bd723\") "
Nov 02 21:52:52.437040 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.437020    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-openshift-ca" pod "5a65a47e-f350-4381-90d6-7aab121bd723" (UID: "5a65a47e-f350-4381-90d6-7aab121bd723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:52:52.437150 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.437135    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:52.438698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.438676    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-xks-ca" pod "5a65a47e-f350-4381-90d6-7aab121bd723" (UID: "5a65a47e-f350-4381-90d6-7aab121bd723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:52:52.438829 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.438811    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a65a47e-f350-4381-90d6-7aab121bd723-kube-api-access-tvfn6" pod "5a65a47e-f350-4381-90d6-7aab121bd723" (UID: "5a65a47e-f350-4381-90d6-7aab121bd723"). InnerVolumeSpecName "kube-api-access-tvfn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:52:52.538125 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.538074    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvfn6\" (UniqueName: \"kubernetes.io/projected/5a65a47e-f350-4381-90d6-7aab121bd723-kube-api-access-tvfn6\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:52.538303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:52.538147    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a65a47e-f350-4381-90d6-7aab121bd723-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:52:53.173562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:53.173530    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"
Nov 02 21:52:53.173769 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:53.173552    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq" event={"ID":"5a65a47e-f350-4381-90d6-7aab121bd723","Type":"ContainerDied","Data":"839790f2b6ab67cd4f0db25bca77d7df2c4456e65749e1026efea7259dbf3be5"}
Nov 02 21:52:53.173769 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:53.173636    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="839790f2b6ab67cd4f0db25bca77d7df2c4456e65749e1026efea7259dbf3be5"
Nov 02 21:52:53.425132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:53.424198    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 02 21:52:53.425132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:53.424256    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 02 21:52:54.422133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:54.416452    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 02 21:52:54.422133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:54.416524    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 02 21:52:55.418133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.412788    4528 patch_prober.go:29] interesting pod/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Nov 02 21:52:55.418133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.412856    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Nov 02 21:52:55.426128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.424240    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"]
Nov 02 21:52:55.426128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.425061    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5a65a47e-f350-4381-90d6-7aab121bd723" containerName="revoke-keys"
Nov 02 21:52:55.426128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.425079    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a65a47e-f350-4381-90d6-7aab121bd723" containerName="revoke-keys"
Nov 02 21:52:55.426128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.425094    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9c91b35c-d00f-4dbb-a601-b0f865238201" containerName="revoke-keys"
Nov 02 21:52:55.426128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.425122    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c91b35c-d00f-4dbb-a601-b0f865238201" containerName="revoke-keys"
Nov 02 21:52:55.426128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.425293    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a65a47e-f350-4381-90d6-7aab121bd723" containerName="revoke-keys"
Nov 02 21:52:55.426128 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.425308    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c91b35c-d00f-4dbb-a601-b0f865238201" containerName="revoke-keys"
Nov 02 21:52:55.430356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.429686    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:52:55.460433 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.460405    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"]
Nov 02 21:52:55.519147 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.516503    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-97995b\""
Nov 02 21:52:55.519147 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.516771    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-97995b-dockercfg-p7c66\""
Nov 02 21:52:55.602144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.601858    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzzw\" (UniqueName: \"kubernetes.io/projected/389df98a-aff3-4a56-ab4a-998ef60e2bae-kube-api-access-kzzzw\") pod \"payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f\" (UID: \"389df98a-aff3-4a56-ab4a-998ef60e2bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:52:55.602144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.601987    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389df98a-aff3-4a56-ab4a-998ef60e2bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f\" (UID: \"389df98a-aff3-4a56-ab4a-998ef60e2bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:52:55.710173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.705974    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzzw\" (UniqueName: \"kubernetes.io/projected/389df98a-aff3-4a56-ab4a-998ef60e2bae-kube-api-access-kzzzw\") pod \"payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f\" (UID: \"389df98a-aff3-4a56-ab4a-998ef60e2bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:52:55.710173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.706090    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389df98a-aff3-4a56-ab4a-998ef60e2bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f\" (UID: \"389df98a-aff3-4a56-ab4a-998ef60e2bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:52:55.710173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.706994    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389df98a-aff3-4a56-ab4a-998ef60e2bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f\" (UID: \"389df98a-aff3-4a56-ab4a-998ef60e2bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:52:55.772476 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:55.772440    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzzw\" (UniqueName: \"kubernetes.io/projected/389df98a-aff3-4a56-ab4a-998ef60e2bae-kube-api-access-kzzzw\") pod \"payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f\" (UID: \"389df98a-aff3-4a56-ab4a-998ef60e2bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:52:56.043889 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:56.043854    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:52:56.415698 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:56.415670    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:52:56.416166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:56.416144    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:52:56.416970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:56.416949    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:52:56.418247 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:52:56.418221    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389df98a_aff3_4a56_ab4a_998ef60e2bae.slice/crio-3aaf43b07ef459785542b0a181f53642c3dcb9ec09516ad5f733a03dff95dc49 WatchSource:0}: Error finding container 3aaf43b07ef459785542b0a181f53642c3dcb9ec09516ad5f733a03dff95dc49: Status 404 returned error can't find the container with id 3aaf43b07ef459785542b0a181f53642c3dcb9ec09516ad5f733a03dff95dc49
Nov 02 21:52:56.421569 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:56.421550    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"]
Nov 02 21:52:57.265828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:57.255088    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f" event={"ID":"389df98a-aff3-4a56-ab4a-998ef60e2bae","Type":"ContainerStarted","Data":"b3eee50480adb329d0b3aecaa3b9a36e88a7a9838971adefda08ea161b4bf8f9"}
Nov 02 21:52:57.265828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:57.265332    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:52:57.265828 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:57.265383    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f" event={"ID":"389df98a-aff3-4a56-ab4a-998ef60e2bae","Type":"ContainerStarted","Data":"3aaf43b07ef459785542b0a181f53642c3dcb9ec09516ad5f733a03dff95dc49"}
Nov 02 21:52:57.373607 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:52:57.373547    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f" podStartSLOduration=2.373530401 podStartE2EDuration="2.373530401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:57.368838787 +0000 UTC m=+2664.632375116" watchObservedRunningTime="2026-11-02 21:52:57.373530401 +0000 UTC m=+2664.637066721"
Nov 02 21:53:01.065274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.061017    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"]
Nov 02 21:53:01.089432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.089401    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.162514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.162465    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"]
Nov 02 21:53:01.169434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.165989    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-data\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.169434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.166067    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-certs\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.169434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.166098    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-credential-socket\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.169434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.166168    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbh4\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-kube-api-access-mgbh4\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.169434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.166257    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33afd9e-0e68-4231-910d-28348af5ab93-istio-podinfo\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.169434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.166285    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33afd9e-0e68-4231-910d-28348af5ab93-istiod-ca-cert\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.169434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.166324    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-socket\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.169434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.166384    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-envoy\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.169434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.166410    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-istio-token\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.195816 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.195785    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-7cde84-openshift-default-dockercfg-98fzn\""
Nov 02 21:53:01.267896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.267860    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-data\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.268097 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.267936    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-certs\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.268097 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.267962    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-credential-socket\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.268097 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.268013    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbh4\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-kube-api-access-mgbh4\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.268325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.268119    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33afd9e-0e68-4231-910d-28348af5ab93-istio-podinfo\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.268325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.268154    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33afd9e-0e68-4231-910d-28348af5ab93-istiod-ca-cert\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.268325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.268196    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-socket\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.268325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.268264    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-envoy\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.268325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.268294    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-istio-token\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.269314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.268601    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-data\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.269770 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.269744    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33afd9e-0e68-4231-910d-28348af5ab93-istiod-ca-cert\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.271050 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.271029    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-certs\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.271429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.271289    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-credential-socket\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.271567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.271383    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-socket\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.294131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.290887    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-envoy\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.315134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.314758    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33afd9e-0e68-4231-910d-28348af5ab93-istio-podinfo\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.331533 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.331402    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbh4\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-kube-api-access-mgbh4\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.332543 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.332510    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-istio-token\") pod \"e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") " pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.420424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.420400    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:01.586143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.586041    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"]
Nov 02 21:53:01.590589 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.590561    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"]
Nov 02 21:53:01.590793 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.590766    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.595243 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.595224    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.643733 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.643711    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d6cb38-openshift-default-dockercfg-mbhbl\""
Nov 02 21:53:01.674223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674192    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.674373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674243    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqtf\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-kube-api-access-xvqtf\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.674373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674351    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.674472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674415    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.674557 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674538    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.674635 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674618    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4467661-9949-48a7-8d51-512c1abf63f4-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.674689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674673    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.674865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674778    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.674865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674830    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.674865 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674861    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.675019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.674933    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.675069 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.675052    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4467661-9949-48a7-8d51-512c1abf63f4-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.675165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.675147    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8btfb\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-kube-api-access-8btfb\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.675216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.675201    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.675255 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.675239    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.675292 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.675266    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.675329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.675295    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.675381 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.675346    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.703726 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.703697    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"]
Nov 02 21:53:01.708672 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.708641    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"]
Nov 02 21:53:01.776875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.776845    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8btfb\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-kube-api-access-8btfb\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.776889    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.776923    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.776944    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.776964    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.776985    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777047    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777080    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqtf\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-kube-api-access-xvqtf\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777138    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777164    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777186    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777257    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4467661-9949-48a7-8d51-512c1abf63f4-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777304    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777356    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777422    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777416    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777439    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777452    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777463    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777476    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777500    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.777798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777566    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4467661-9949-48a7-8d51-512c1abf63f4-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.777798 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777760    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.778315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.777825    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.778315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.778022    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.778315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.778141    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.778315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.778136    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.778315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.778273    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4467661-9949-48a7-8d51-512c1abf63f4-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.779597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.779572    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.779808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.779790    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.780051 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.780033    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.780247 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.780228    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4467661-9949-48a7-8d51-512c1abf63f4-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.806446 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:53:01.806420    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33afd9e_0e68_4231_910d_28348af5ab93.slice/crio-9437d2ed0c81e928b78a999d2c96f8a4495eb3acfeab95362e90f00fd93b749d WatchSource:0}: Error finding container 9437d2ed0c81e928b78a999d2c96f8a4495eb3acfeab95362e90f00fd93b749d: Status 404 returned error can't find the container with id 9437d2ed0c81e928b78a999d2c96f8a4495eb3acfeab95362e90f00fd93b749d
Nov 02 21:53:01.809003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.808974    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:53:01.809090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.809055    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:53:01.809090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.809083    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:53:01.811732 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.810472    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"]
Nov 02 21:53:01.836933 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.836883    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8btfb\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-kube-api-access-8btfb\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.837261 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.837243    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.837399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.837378    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:01.837521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.837507    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqtf\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-kube-api-access-xvqtf\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.908989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.908968    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:01.915008 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:01.914989    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:02.292231 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:53:02.292166    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdb0c3b_9b7b_43d0_aed7_fc8d516a4cb0.slice/crio-38ad08de975a12fd47ed5f45a54fcd050a47a812d1120be5728d7385b2be1a0f WatchSource:0}: Error finding container 38ad08de975a12fd47ed5f45a54fcd050a47a812d1120be5728d7385b2be1a0f: Status 404 returned error can't find the container with id 38ad08de975a12fd47ed5f45a54fcd050a47a812d1120be5728d7385b2be1a0f
Nov 02 21:53:02.295038 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.295003    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:53:02.295196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.295087    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:53:02.295196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.295142    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:53:02.297195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.297174    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"]
Nov 02 21:53:02.342314 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:53:02.342278    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4467661_9949_48a7_8d51_512c1abf63f4.slice/crio-b178be0b816453ab4a716fa94e9b0031779036964f6f217afbfa1198cee3635f WatchSource:0}: Error finding container b178be0b816453ab4a716fa94e9b0031779036964f6f217afbfa1198cee3635f: Status 404 returned error can't find the container with id b178be0b816453ab4a716fa94e9b0031779036964f6f217afbfa1198cee3635f
Nov 02 21:53:02.345061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.345027    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:53:02.345176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.345093    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:53:02.345176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.345156    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:53:02.346220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.346200    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"]
Nov 02 21:53:02.356631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.356603    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" event={"ID":"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0","Type":"ContainerStarted","Data":"38ad08de975a12fd47ed5f45a54fcd050a47a812d1120be5728d7385b2be1a0f"}
Nov 02 21:53:02.357918 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.357892    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" event={"ID":"c4467661-9949-48a7-8d51-512c1abf63f4","Type":"ContainerStarted","Data":"b178be0b816453ab4a716fa94e9b0031779036964f6f217afbfa1198cee3635f"}
Nov 02 21:53:02.359700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.359681    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" event={"ID":"b33afd9e-0e68-4231-910d-28348af5ab93","Type":"ContainerStarted","Data":"c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b"}
Nov 02 21:53:02.359817 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.359716    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" event={"ID":"b33afd9e-0e68-4231-910d-28348af5ab93","Type":"ContainerStarted","Data":"9437d2ed0c81e928b78a999d2c96f8a4495eb3acfeab95362e90f00fd93b749d"}
Nov 02 21:53:02.420957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.420905    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:02.426593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.426573    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:02.486005 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:02.483622    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" podStartSLOduration=2.483602115 podStartE2EDuration="2.483602115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:02.475987414 +0000 UTC m=+2669.739523733" watchObservedRunningTime="2026-11-02 21:53:02.483602115 +0000 UTC m=+2669.747138433"
Nov 02 21:53:03.369462 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.369424    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" event={"ID":"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0","Type":"ContainerStarted","Data":"e27ab832b6fd64ac6e183b3097bf720364ab09042a3f9d954c6797a231c24b9a"}
Nov 02 21:53:03.371064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.371024    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" event={"ID":"c4467661-9949-48a7-8d51-512c1abf63f4","Type":"ContainerStarted","Data":"d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754"}
Nov 02 21:53:03.371302 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.371273    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:03.372279 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.372259    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:53:03.490901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.490853    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" podStartSLOduration=2.490841158 podStartE2EDuration="2.490841158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:03.489775859 +0000 UTC m=+2670.753312175" watchObservedRunningTime="2026-11-02 21:53:03.490841158 +0000 UTC m=+2670.754377542"
Nov 02 21:53:03.766317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.765351    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" podStartSLOduration=2.765326655 podStartE2EDuration="2.765326655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:03.758625341 +0000 UTC m=+2671.022161672" watchObservedRunningTime="2026-11-02 21:53:03.765326655 +0000 UTC m=+2671.028862973"
Nov 02 21:53:03.912888 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.911473    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:03.928135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.927249    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 21:53:03.928135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.927314    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 21:53:03.928734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.928710    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:03.939139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.937237    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 21:53:03.939139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:03.937295    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 21:53:04.955254 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:04.955213    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 21:53:04.955743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:04.955289    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 21:53:04.955743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:04.955216    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 21:53:04.955743 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:04.955366    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 21:53:05.923133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:05.914467    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 21:53:05.923133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:05.914536    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 21:53:05.925473 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:05.925446    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 21:53:05.925641 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:05.925617    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 21:53:06.966411 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:06.966379    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 21:53:06.973898 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:06.973842    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 21:53:06.974650 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:06.967927    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 21:53:06.974864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:06.974839    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 21:53:07.294133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.291728    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"]
Nov 02 21:53:07.487138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.486825    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"]
Nov 02 21:53:07.487319 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.487290    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:07.552211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.548049    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-6e535d-dockercfg-5lm47\""
Nov 02 21:53:07.552211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.551808    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb989\" (UniqueName: \"kubernetes.io/projected/cfa258a7-12ac-4547-80ce-3e4826883324-kube-api-access-bb989\") pod \"payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t\" (UID: \"cfa258a7-12ac-4547-80ce-3e4826883324\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:07.552211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.551858    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa258a7-12ac-4547-80ce-3e4826883324-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t\" (UID: \"cfa258a7-12ac-4547-80ce-3e4826883324\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:07.598256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.598232    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-6e535d\""
Nov 02 21:53:07.654449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.654407    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb989\" (UniqueName: \"kubernetes.io/projected/cfa258a7-12ac-4547-80ce-3e4826883324-kube-api-access-bb989\") pod \"payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t\" (UID: \"cfa258a7-12ac-4547-80ce-3e4826883324\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:07.654658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.654467    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa258a7-12ac-4547-80ce-3e4826883324-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t\" (UID: \"cfa258a7-12ac-4547-80ce-3e4826883324\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:07.655406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.655383    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa258a7-12ac-4547-80ce-3e4826883324-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t\" (UID: \"cfa258a7-12ac-4547-80ce-3e4826883324\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:07.750151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.746246    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb989\" (UniqueName: \"kubernetes.io/projected/cfa258a7-12ac-4547-80ce-3e4826883324-kube-api-access-bb989\") pod \"payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t\" (UID: \"cfa258a7-12ac-4547-80ce-3e4826883324\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:07.819188 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.815322    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:07.910989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.910743    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 21:53:07.944269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.915364    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 21:53:07.944269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.915412    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 21:53:07.944269 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:07.923204    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 21:53:08.266401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:08.266372    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"]
Nov 02 21:53:08.283689 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:08.283668    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 21:53:08.515195 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:08.514623    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t" event={"ID":"cfa258a7-12ac-4547-80ce-3e4826883324","Type":"ContainerStarted","Data":"6860ce65d3f71437fa11bba9846d05f19ed34cf9f86aeea58df227ac33c4381e"}
Nov 02 21:53:08.912191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:08.910512    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 21:53:08.912191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:08.910570    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 21:53:08.926499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:08.926470    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 21:53:08.926741 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:08.926716    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 21:53:09.524963 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:09.524926    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t" event={"ID":"cfa258a7-12ac-4547-80ce-3e4826883324","Type":"ContainerStarted","Data":"63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837"}
Nov 02 21:53:09.525410 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:09.525073    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:09.648694 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:09.648632    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t" podStartSLOduration=2.648612274 podStartE2EDuration="2.648612274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:09.641785365 +0000 UTC m=+2676.905321683" watchObservedRunningTime="2026-11-02 21:53:09.648612274 +0000 UTC m=+2676.912148592"
Nov 02 21:53:09.909896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:09.909814    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 21:53:09.909896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:09.909879    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 21:53:10.179875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:10.179787    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:10.181070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:10.181013    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:10.181439 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:10.181420    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 21:53:10.914730 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:10.914704    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:10.915169 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:10.915044    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:10.915687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:10.915667    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 21:53:11.387168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:11.387137    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:53:11.839383 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:11.839348    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"]
Nov 02 21:53:11.843975 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:11.843955    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:11.900521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:11.900475    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-7cde84\""
Nov 02 21:53:11.904710 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:11.904623    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-7cde84-dockercfg-9qwlf\""
Nov 02 21:53:11.906504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:11.906474    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"]
Nov 02 21:53:12.059104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:12.059064    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/b0f4ec96-a910-44dd-aab7-245f8a0471f4-kube-api-access-qpwks\") pod \"payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g\" (UID: \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:12.064151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:12.063884    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f4ec96-a910-44dd-aab7-245f8a0471f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g\" (UID: \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:12.195175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:12.194238    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f4ec96-a910-44dd-aab7-245f8a0471f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g\" (UID: \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:12.195175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:12.194333    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/b0f4ec96-a910-44dd-aab7-245f8a0471f4-kube-api-access-qpwks\") pod \"payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g\" (UID: \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:12.216143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:12.195802    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f4ec96-a910-44dd-aab7-245f8a0471f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g\" (UID: \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:12.275019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:12.274918    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/b0f4ec96-a910-44dd-aab7-245f8a0471f4-kube-api-access-qpwks\") pod \"payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g\" (UID: \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:12.467285 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:12.467200    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:13.493209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:13.492197    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"]
Nov 02 21:53:13.732070 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:13.732031    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g" event={"ID":"b0f4ec96-a910-44dd-aab7-245f8a0471f4","Type":"ContainerStarted","Data":"0940ad00b8c0a0a021421f060c39125752cd1f50d324dadfa25cc43f9866d6e5"}
Nov 02 21:53:14.298135 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:53:14.287849    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:53:14.793131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:14.792228    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:53:14.823144 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:14.818397    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g" event={"ID":"b0f4ec96-a910-44dd-aab7-245f8a0471f4","Type":"ContainerStarted","Data":"2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33"}
Nov 02 21:53:14.823849 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:14.823785    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:14.833127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:14.828198    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:53:20.546083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:20.546057    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 21:53:20.662442 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:20.662391    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g" podStartSLOduration=9.662378715 podStartE2EDuration="9.662378715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:14.980861737 +0000 UTC m=+2682.244398055" watchObservedRunningTime="2026-11-02 21:53:20.662378715 +0000 UTC m=+2687.925915028"
Nov 02 21:53:22.760183 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:22.760142    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"]
Nov 02 21:53:26.834028 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:26.833996    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 21:53:34.904369 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:34.904338    4528 scope.go:122] "RemoveContainer" containerID="32a26e267e1b554d222a27db328345dc801047320ad6bca06a13541fc2e47387"
Nov 02 21:53:37.134403 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:37.134372    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"]
Nov 02 21:53:37.231857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:37.231823    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"]
Nov 02 21:53:37.780078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:37.780044    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq" podUID="ff8babdc-4005-4220-9023-a91d3b34fd2e" containerName="main" containerID="cri-o://070881ad63ba9b36dd0e71dd7c1805f489250b1ffd0feec0229b2532b4453d38" gracePeriod=45
Nov 02 21:53:37.985147 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:37.982929    4528 generic.go:356] "Generic (PLEG): container finished" podID="ff8babdc-4005-4220-9023-a91d3b34fd2e" containerID="070881ad63ba9b36dd0e71dd7c1805f489250b1ffd0feec0229b2532b4453d38" exitCode=0
Nov 02 21:53:37.985147 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:37.983191    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq" event={"ID":"ff8babdc-4005-4220-9023-a91d3b34fd2e","Type":"ContainerDied","Data":"070881ad63ba9b36dd0e71dd7c1805f489250b1ffd0feec0229b2532b4453d38"}
Nov 02 21:53:38.032525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.032464    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:53:38.154096 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.154065    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8babdc-4005-4220-9023-a91d3b34fd2e-kube-api-access-f8gdk\" (UniqueName: \"kubernetes.io/projected/ff8babdc-4005-4220-9023-a91d3b34fd2e-kube-api-access-f8gdk\") pod \"ff8babdc-4005-4220-9023-a91d3b34fd2e\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") "
Nov 02 21:53:38.154599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.154134    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-home\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-home\") pod \"ff8babdc-4005-4220-9023-a91d3b34fd2e\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") "
Nov 02 21:53:38.154599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.154245    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff8babdc-4005-4220-9023-a91d3b34fd2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8babdc-4005-4220-9023-a91d3b34fd2e-tls-certs\") pod \"ff8babdc-4005-4220-9023-a91d3b34fd2e\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") "
Nov 02 21:53:38.154599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.154329    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-dshm\") pod \"ff8babdc-4005-4220-9023-a91d3b34fd2e\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") "
Nov 02 21:53:38.154599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.154363    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-model-cache\") pod \"ff8babdc-4005-4220-9023-a91d3b34fd2e\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") "
Nov 02 21:53:38.154599 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.154478    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-tmp-dir\") pod \"ff8babdc-4005-4220-9023-a91d3b34fd2e\" (UID: \"ff8babdc-4005-4220-9023-a91d3b34fd2e\") "
Nov 02 21:53:38.155303 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.155256    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-model-cache" pod "ff8babdc-4005-4220-9023-a91d3b34fd2e" (UID: "ff8babdc-4005-4220-9023-a91d3b34fd2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:53:38.155677 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.155615    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-tmp-dir" pod "ff8babdc-4005-4220-9023-a91d3b34fd2e" (UID: "ff8babdc-4005-4220-9023-a91d3b34fd2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:53:38.155772 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.155717    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-home" pod "ff8babdc-4005-4220-9023-a91d3b34fd2e" (UID: "ff8babdc-4005-4220-9023-a91d3b34fd2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:53:38.156826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.156801    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8babdc-4005-4220-9023-a91d3b34fd2e-kube-api-access-f8gdk" pod "ff8babdc-4005-4220-9023-a91d3b34fd2e" (UID: "ff8babdc-4005-4220-9023-a91d3b34fd2e"). InnerVolumeSpecName "kube-api-access-f8gdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:53:38.156943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.156877    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff8babdc-4005-4220-9023-a91d3b34fd2e-tls-certs" pod "ff8babdc-4005-4220-9023-a91d3b34fd2e" (UID: "ff8babdc-4005-4220-9023-a91d3b34fd2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:53:38.157205 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.157184    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-dshm" pod "ff8babdc-4005-4220-9023-a91d3b34fd2e" (UID: "ff8babdc-4005-4220-9023-a91d3b34fd2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:53:38.262312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.262281    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8babdc-4005-4220-9023-a91d3b34fd2e-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:38.262312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.262310    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:38.262312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.262322    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:38.262632 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.262334    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:38.262632 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.262347    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8gdk\" (UniqueName: \"kubernetes.io/projected/ff8babdc-4005-4220-9023-a91d3b34fd2e-kube-api-access-f8gdk\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:38.262632 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.262360    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff8babdc-4005-4220-9023-a91d3b34fd2e-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:38.957960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.957907    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq" podUID="ff8babdc-4005-4220-9023-a91d3b34fd2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:53:38.991207 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.991183    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"
Nov 02 21:53:38.991374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.991202    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq" event={"ID":"ff8babdc-4005-4220-9023-a91d3b34fd2e","Type":"ContainerDied","Data":"7fd1c6a26c1d830238929da2f0315d59980f7481fbc5b8b89f764881ec6f4f74"}
Nov 02 21:53:38.991374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:38.991252    4528 scope.go:122] "RemoveContainer" containerID="070881ad63ba9b36dd0e71dd7c1805f489250b1ffd0feec0229b2532b4453d38"
Nov 02 21:53:39.200504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:39.200430    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"]
Nov 02 21:53:39.203986 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:39.203963    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-aa8aff/praxis-route-aa8aff-kserve-868649bc55-92kmq"]
Nov 02 21:53:40.913703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:40.913673    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8babdc-4005-4220-9023-a91d3b34fd2e" path="/var/lib/kubelet/pods/ff8babdc-4005-4220-9023-a91d3b34fd2e/volumes"
Nov 02 21:53:42.137271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.137199    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms" podUID="8ce62acb-45af-4da8-a5ae-fa6d381144a6" containerName="payload-processing" containerID="cri-o://795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2" gracePeriod=25
Nov 02 21:53:42.232972 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.232934    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw" podUID="31213d06-22c3-430f-a99e-a54d475d1597" containerName="payload-pre-processing" containerID="cri-o://d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719" gracePeriod=25
Nov 02 21:53:42.496914 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.496888    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:53:42.522491 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.522469    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:53:42.615872 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.615844    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31213d06-22c3-430f-a99e-a54d475d1597-kube-api-access-x9p2v\" (UniqueName: \"kubernetes.io/projected/31213d06-22c3-430f-a99e-a54d475d1597-kube-api-access-x9p2v\") pod \"31213d06-22c3-430f-a99e-a54d475d1597\" (UID: \"31213d06-22c3-430f-a99e-a54d475d1597\") "
Nov 02 21:53:42.616052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.615896    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce62acb-45af-4da8-a5ae-fa6d381144a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce62acb-45af-4da8-a5ae-fa6d381144a6-plugins-config-volume\") pod \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\" (UID: \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\") "
Nov 02 21:53:42.616052 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.616035    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31213d06-22c3-430f-a99e-a54d475d1597-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31213d06-22c3-430f-a99e-a54d475d1597-plugins-config-volume\") pod \"31213d06-22c3-430f-a99e-a54d475d1597\" (UID: \"31213d06-22c3-430f-a99e-a54d475d1597\") "
Nov 02 21:53:42.616192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.616121    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce62acb-45af-4da8-a5ae-fa6d381144a6-kube-api-access-qh7c5\" (UniqueName: \"kubernetes.io/projected/8ce62acb-45af-4da8-a5ae-fa6d381144a6-kube-api-access-qh7c5\") pod \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\" (UID: \"8ce62acb-45af-4da8-a5ae-fa6d381144a6\") "
Nov 02 21:53:42.616419 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.616373    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce62acb-45af-4da8-a5ae-fa6d381144a6-plugins-config-volume" pod "8ce62acb-45af-4da8-a5ae-fa6d381144a6" (UID: "8ce62acb-45af-4da8-a5ae-fa6d381144a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:53:42.616518 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.616416    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31213d06-22c3-430f-a99e-a54d475d1597-plugins-config-volume" pod "31213d06-22c3-430f-a99e-a54d475d1597" (UID: "31213d06-22c3-430f-a99e-a54d475d1597"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:53:42.616684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.616664    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce62acb-45af-4da8-a5ae-fa6d381144a6-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:42.616767 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.616686    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31213d06-22c3-430f-a99e-a54d475d1597-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:42.617992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.617971    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31213d06-22c3-430f-a99e-a54d475d1597-kube-api-access-x9p2v" pod "31213d06-22c3-430f-a99e-a54d475d1597" (UID: "31213d06-22c3-430f-a99e-a54d475d1597"). InnerVolumeSpecName "kube-api-access-x9p2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:53:42.618232 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.618203    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce62acb-45af-4da8-a5ae-fa6d381144a6-kube-api-access-qh7c5" pod "8ce62acb-45af-4da8-a5ae-fa6d381144a6" (UID: "8ce62acb-45af-4da8-a5ae-fa6d381144a6"). InnerVolumeSpecName "kube-api-access-qh7c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:53:42.717723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.717653    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh7c5\" (UniqueName: \"kubernetes.io/projected/8ce62acb-45af-4da8-a5ae-fa6d381144a6-kube-api-access-qh7c5\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:42.717723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:42.717679    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9p2v\" (UniqueName: \"kubernetes.io/projected/31213d06-22c3-430f-a99e-a54d475d1597-kube-api-access-x9p2v\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:43.019499 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.019461    4528 generic.go:356] "Generic (PLEG): container finished" podID="8ce62acb-45af-4da8-a5ae-fa6d381144a6" containerID="795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2" exitCode=0
Nov 02 21:53:43.019669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.019528    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"
Nov 02 21:53:43.019669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.019545    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms" event={"ID":"8ce62acb-45af-4da8-a5ae-fa6d381144a6","Type":"ContainerDied","Data":"795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2"}
Nov 02 21:53:43.019669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.019593    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms" event={"ID":"8ce62acb-45af-4da8-a5ae-fa6d381144a6","Type":"ContainerDied","Data":"b0ebcb6d7e3038d8ff9230674ce8314073fca5d2c7f5f7b5c0cfb529524f6685"}
Nov 02 21:53:43.019669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.019614    4528 scope.go:122] "RemoveContainer" containerID="795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2"
Nov 02 21:53:43.020994 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.020963    4528 generic.go:356] "Generic (PLEG): container finished" podID="31213d06-22c3-430f-a99e-a54d475d1597" containerID="d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719" exitCode=0
Nov 02 21:53:43.021148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.021026    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"
Nov 02 21:53:43.021148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.021049    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw" event={"ID":"31213d06-22c3-430f-a99e-a54d475d1597","Type":"ContainerDied","Data":"d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719"}
Nov 02 21:53:43.021148 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.021088    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw" event={"ID":"31213d06-22c3-430f-a99e-a54d475d1597","Type":"ContainerDied","Data":"b92af4e322494d0a2ad9da0f54565ced5a36881d5cda2e3a53f0e7a7b7eaea61"}
Nov 02 21:53:43.033261 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.033245    4528 scope.go:122] "RemoveContainer" containerID="795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2"
Nov 02 21:53:43.033538 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:53:43.033508    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2\": container with ID starting with 795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2 not found: ID does not exist" containerID="795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2"
Nov 02 21:53:43.033642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.033542    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2"} err="failed to get container status \"795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2\": rpc error: code = NotFound desc = could not find container \"795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2\": container with ID starting with 795965b65642ab3191e886c683adbe764d4f4376b860d8e12f0c551a191cd2c2 not found: ID does not exist"
Nov 02 21:53:43.033642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.033558    4528 scope.go:122] "RemoveContainer" containerID="d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719"
Nov 02 21:53:43.042236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.042220    4528 scope.go:122] "RemoveContainer" containerID="d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719"
Nov 02 21:53:43.042503 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:53:43.042486    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719\": container with ID starting with d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719 not found: ID does not exist" containerID="d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719"
Nov 02 21:53:43.042556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.042508    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719"} err="failed to get container status \"d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719\": rpc error: code = NotFound desc = could not find container \"d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719\": container with ID starting with d71249eda31dc31d1dca33666f8f2499ee4c090ac3f33e0d8a5d4ff414ec1719 not found: ID does not exist"
Nov 02 21:53:43.184958 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.184934    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"]
Nov 02 21:53:43.188656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.188638    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eabda2-f546f6ccc-xxqms"]
Nov 02 21:53:43.299453 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.299379    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"]
Nov 02 21:53:43.302953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:43.302932    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eabda2-64f97db6cf-vz6qw"]
Nov 02 21:53:44.112528 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:53:44.112485    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:53:44.913610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:44.913577    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31213d06-22c3-430f-a99e-a54d475d1597" path="/var/lib/kubelet/pods/31213d06-22c3-430f-a99e-a54d475d1597/volumes"
Nov 02 21:53:44.914045 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:44.914030    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce62acb-45af-4da8-a5ae-fa6d381144a6" path="/var/lib/kubelet/pods/8ce62acb-45af-4da8-a5ae-fa6d381144a6/volumes"
Nov 02 21:53:51.278370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:51.278332    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"]
Nov 02 21:53:52.109524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.109483    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"]
Nov 02 21:53:52.110088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.110069    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8ce62acb-45af-4da8-a5ae-fa6d381144a6" containerName="payload-processing"
Nov 02 21:53:52.110231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.110089    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce62acb-45af-4da8-a5ae-fa6d381144a6" containerName="payload-processing"
Nov 02 21:53:52.110231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.110135    4528 cpu_manager.go:414] "containerMap: removing container" podUID="31213d06-22c3-430f-a99e-a54d475d1597" containerName="payload-pre-processing"
Nov 02 21:53:52.110231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.110144    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31213d06-22c3-430f-a99e-a54d475d1597" containerName="payload-pre-processing"
Nov 02 21:53:52.110231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.110175    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ff8babdc-4005-4220-9023-a91d3b34fd2e" containerName="main"
Nov 02 21:53:52.110231 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.110182    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8babdc-4005-4220-9023-a91d3b34fd2e" containerName="main"
Nov 02 21:53:52.110485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.110334    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce62acb-45af-4da8-a5ae-fa6d381144a6" containerName="payload-processing"
Nov 02 21:53:52.110485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.110349    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="31213d06-22c3-430f-a99e-a54d475d1597" containerName="payload-pre-processing"
Nov 02 21:53:52.110485 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.110362    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8babdc-4005-4220-9023-a91d3b34fd2e" containerName="main"
Nov 02 21:53:52.114740 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.114721    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.167399 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.167372    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:53:52.209422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.209395    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzv6\" (UniqueName: \"kubernetes.io/projected/0f992866-c294-4889-a086-8b00a6fae157-kube-api-access-4lzv6\") pod \"maas-api-revoke-sub-b67e1fda-lfvvq\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.209572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.209428    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f992866-c294-4889-a086-8b00a6fae157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b67e1fda-lfvvq\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.209572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.209467    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f992866-c294-4889-a086-8b00a6fae157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b67e1fda-lfvvq\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.213424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.213402    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:53:52.213424 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.213420    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:53:52.213575 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.213530    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:53:52.217307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.217288    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"]
Nov 02 21:53:52.311158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.311129    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzv6\" (UniqueName: \"kubernetes.io/projected/0f992866-c294-4889-a086-8b00a6fae157-kube-api-access-4lzv6\") pod \"maas-api-revoke-sub-b67e1fda-lfvvq\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.311537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.311172    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f992866-c294-4889-a086-8b00a6fae157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b67e1fda-lfvvq\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.311537 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.311204    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f992866-c294-4889-a086-8b00a6fae157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b67e1fda-lfvvq\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.311857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.311829    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f992866-c294-4889-a086-8b00a6fae157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b67e1fda-lfvvq\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.314779 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.314755    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f992866-c294-4889-a086-8b00a6fae157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b67e1fda-lfvvq\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.371163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.371088    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzv6\" (UniqueName: \"kubernetes.io/projected/0f992866-c294-4889-a086-8b00a6fae157-kube-api-access-4lzv6\") pod \"maas-api-revoke-sub-b67e1fda-lfvvq\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.429431 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.429395    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:52.851143 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:52.848135    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"]
Nov 02 21:53:53.138826 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:53.138791    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq" event={"ID":"0f992866-c294-4889-a086-8b00a6fae157","Type":"ContainerStarted","Data":"a94d75044f0c2840ed59b8ed0e2f84eb4509ce370d81ac75b430eaf52aa9e937"}
Nov 02 21:53:54.147409 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:54.147321    4528 generic.go:356] "Generic (PLEG): container finished" podID="0f992866-c294-4889-a086-8b00a6fae157" containerID="7a203aa3d8b4c42e0bab77a8f42ffa40afd15340126597465f784fee622882eb" exitCode=0
Nov 02 21:53:54.147768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:54.147407    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq" event={"ID":"0f992866-c294-4889-a086-8b00a6fae157","Type":"ContainerDied","Data":"7a203aa3d8b4c42e0bab77a8f42ffa40afd15340126597465f784fee622882eb"}
Nov 02 21:53:55.304823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.304802    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:55.356018 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.355991    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f992866-c294-4889-a086-8b00a6fae157-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f992866-c294-4889-a086-8b00a6fae157-maas-api-openshift-ca\") pod \"0f992866-c294-4889-a086-8b00a6fae157\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") "
Nov 02 21:53:55.356237 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.356050    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f992866-c294-4889-a086-8b00a6fae157-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f992866-c294-4889-a086-8b00a6fae157-maas-api-xks-ca\") pod \"0f992866-c294-4889-a086-8b00a6fae157\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") "
Nov 02 21:53:55.356237 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.356150    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f992866-c294-4889-a086-8b00a6fae157-kube-api-access-4lzv6\" (UniqueName: \"kubernetes.io/projected/0f992866-c294-4889-a086-8b00a6fae157-kube-api-access-4lzv6\") pod \"0f992866-c294-4889-a086-8b00a6fae157\" (UID: \"0f992866-c294-4889-a086-8b00a6fae157\") "
Nov 02 21:53:55.356366 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.356341    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f992866-c294-4889-a086-8b00a6fae157-maas-api-openshift-ca" pod "0f992866-c294-4889-a086-8b00a6fae157" (UID: "0f992866-c294-4889-a086-8b00a6fae157"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:53:55.356640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.356618    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f992866-c294-4889-a086-8b00a6fae157-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:55.358088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.358065    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f992866-c294-4889-a086-8b00a6fae157-kube-api-access-4lzv6" pod "0f992866-c294-4889-a086-8b00a6fae157" (UID: "0f992866-c294-4889-a086-8b00a6fae157"). InnerVolumeSpecName "kube-api-access-4lzv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:53:55.358323 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.358305    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f992866-c294-4889-a086-8b00a6fae157-maas-api-xks-ca" pod "0f992866-c294-4889-a086-8b00a6fae157" (UID: "0f992866-c294-4889-a086-8b00a6fae157"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:53:55.457678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.457605    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f992866-c294-4889-a086-8b00a6fae157-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:55.457678 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:55.457629    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lzv6\" (UniqueName: \"kubernetes.io/projected/0f992866-c294-4889-a086-8b00a6fae157-kube-api-access-4lzv6\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:53:56.162794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:56.162770    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"
Nov 02 21:53:56.162992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:56.162762    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq" event={"ID":"0f992866-c294-4889-a086-8b00a6fae157","Type":"ContainerDied","Data":"a94d75044f0c2840ed59b8ed0e2f84eb4509ce370d81ac75b430eaf52aa9e937"}
Nov 02 21:53:56.162992 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:56.162908    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a94d75044f0c2840ed59b8ed0e2f84eb4509ce370d81ac75b430eaf52aa9e937"
Nov 02 21:53:56.279525 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:53:56.279488    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45" podUID="5834a1eb-f3fc-45a9-a974-4717a6b46b8f" containerName="payload-processing" containerID="cri-o://354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944" gracePeriod=25
Nov 02 21:54:02.904193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:02.904147    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45" podUID="5834a1eb-f3fc-45a9-a974-4717a6b46b8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:54:07.574021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:07.573982    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:54:12.904827 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:12.904785    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45" podUID="5834a1eb-f3fc-45a9-a974-4717a6b46b8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 21:54:14.112331 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:54:14.112294    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:54:16.271970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:16.271932    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"]
Nov 02 21:54:16.272808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:16.272772    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy" containerID="cri-o://25171310f502d4f2cf575d94f65652e88b6c7233930515e9dea49c02ebebb601" gracePeriod=30
Nov 02 21:54:16.652724 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:16.652702    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:54:16.741335 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:16.741306    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-plugins-config-volume\") pod \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\" (UID: \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\") "
Nov 02 21:54:16.741553 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:16.741348    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-kube-api-access-hzlz5\" (UniqueName: \"kubernetes.io/projected/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-kube-api-access-hzlz5\") pod \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\" (UID: \"5834a1eb-f3fc-45a9-a974-4717a6b46b8f\") "
Nov 02 21:54:16.741776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:16.741750    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-plugins-config-volume" pod "5834a1eb-f3fc-45a9-a974-4717a6b46b8f" (UID: "5834a1eb-f3fc-45a9-a974-4717a6b46b8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:54:16.741970 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:16.741952    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:16.743463 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:16.743444    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-kube-api-access-hzlz5" pod "5834a1eb-f3fc-45a9-a974-4717a6b46b8f" (UID: "5834a1eb-f3fc-45a9-a974-4717a6b46b8f"). InnerVolumeSpecName "kube-api-access-hzlz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:54:16.843511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:16.843439    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzlz5\" (UniqueName: \"kubernetes.io/projected/5834a1eb-f3fc-45a9-a974-4717a6b46b8f-kube-api-access-hzlz5\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:17.380380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:17.380348    4528 generic.go:356] "Generic (PLEG): container finished" podID="5834a1eb-f3fc-45a9-a974-4717a6b46b8f" containerID="354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944" exitCode=0
Nov 02 21:54:17.380796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:17.380393    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"
Nov 02 21:54:17.380796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:17.380437    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45" event={"ID":"5834a1eb-f3fc-45a9-a974-4717a6b46b8f","Type":"ContainerDied","Data":"354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944"}
Nov 02 21:54:17.380796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:17.380488    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45" event={"ID":"5834a1eb-f3fc-45a9-a974-4717a6b46b8f","Type":"ContainerDied","Data":"434caffde51ea971feb4257f0d0b0d64606620f844e921ca7f1fc3de800ec3c8"}
Nov 02 21:54:17.380796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:17.380516    4528 scope.go:122] "RemoveContainer" containerID="354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944"
Nov 02 21:54:17.391428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:17.391403    4528 scope.go:122] "RemoveContainer" containerID="354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944"
Nov 02 21:54:17.391721 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:54:17.391698    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944\": container with ID starting with 354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944 not found: ID does not exist" containerID="354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944"
Nov 02 21:54:17.391839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:17.391725    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944"} err="failed to get container status \"354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944\": rpc error: code = NotFound desc = could not find container \"354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944\": container with ID starting with 354c734d389b30d6a9e27b256885c0b39292590e8caf9f2324c3ec944f586944 not found: ID does not exist"
Nov 02 21:54:17.547722 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:17.547693    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"]
Nov 02 21:54:17.597786 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:17.597757    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-aa8aff-69b45cf765-rsc45"]
Nov 02 21:54:18.914700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:18.914667    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5834a1eb-f3fc-45a9-a974-4717a6b46b8f" path="/var/lib/kubelet/pods/5834a1eb-f3fc-45a9-a974-4717a6b46b8f/volumes"
Nov 02 21:54:21.411510 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.411479    4528 generic.go:356] "Generic (PLEG): container finished" podID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerID="25171310f502d4f2cf575d94f65652e88b6c7233930515e9dea49c02ebebb601" exitCode=0
Nov 02 21:54:21.411939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.411533    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" event={"ID":"8cbf6236-95c3-4847-9650-f5de5af07ad2","Type":"ContainerDied","Data":"25171310f502d4f2cf575d94f65652e88b6c7233930515e9dea49c02ebebb601"}
Nov 02 21:54:21.411939 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.411580    4528 scope.go:122] "RemoveContainer" containerID="78323832e5ffd012c315bf361e38803f14d376ab18f74471d764ca2407edef06"
Nov 02 21:54:21.548659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.548594    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:54:21.599171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599142    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-data\") pod \"8cbf6236-95c3-4847-9650-f5de5af07ad2\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") "
Nov 02 21:54:21.599338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599192    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-certs\") pod \"8cbf6236-95c3-4847-9650-f5de5af07ad2\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") "
Nov 02 21:54:21.599338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599267    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-kube-api-access-jzwtk\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-kube-api-access-jzwtk\") pod \"8cbf6236-95c3-4847-9650-f5de5af07ad2\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") "
Nov 02 21:54:21.599338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599304    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-socket\") pod \"8cbf6236-95c3-4847-9650-f5de5af07ad2\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") "
Nov 02 21:54:21.599338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599326    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbf6236-95c3-4847-9650-f5de5af07ad2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbf6236-95c3-4847-9650-f5de5af07ad2-istiod-ca-cert\") pod \"8cbf6236-95c3-4847-9650-f5de5af07ad2\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") "
Nov 02 21:54:21.599563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599356    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-credential-socket\") pod \"8cbf6236-95c3-4847-9650-f5de5af07ad2\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") "
Nov 02 21:54:21.599563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599403    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-token\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-token\") pod \"8cbf6236-95c3-4847-9650-f5de5af07ad2\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") "
Nov 02 21:54:21.599563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599406    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-data" pod "8cbf6236-95c3-4847-9650-f5de5af07ad2" (UID: "8cbf6236-95c3-4847-9650-f5de5af07ad2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:21.599563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599442    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-certs" pod "8cbf6236-95c3-4847-9650-f5de5af07ad2" (UID: "8cbf6236-95c3-4847-9650-f5de5af07ad2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:21.599563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599492    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-podinfo\") pod \"8cbf6236-95c3-4847-9650-f5de5af07ad2\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") "
Nov 02 21:54:21.599832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599596    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-socket" pod "8cbf6236-95c3-4847-9650-f5de5af07ad2" (UID: "8cbf6236-95c3-4847-9650-f5de5af07ad2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:21.599832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599629    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-envoy\") pod \"8cbf6236-95c3-4847-9650-f5de5af07ad2\" (UID: \"8cbf6236-95c3-4847-9650-f5de5af07ad2\") "
Nov 02 21:54:21.599832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599750    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-credential-socket" pod "8cbf6236-95c3-4847-9650-f5de5af07ad2" (UID: "8cbf6236-95c3-4847-9650-f5de5af07ad2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:21.599832 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.599756    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbf6236-95c3-4847-9650-f5de5af07ad2-istiod-ca-cert" pod "8cbf6236-95c3-4847-9650-f5de5af07ad2" (UID: "8cbf6236-95c3-4847-9650-f5de5af07ad2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:54:21.600220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.600194    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:21.600329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.600221    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbf6236-95c3-4847-9650-f5de5af07ad2-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:21.600329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.600238    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:21.600329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.600252    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:21.600329 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.600266    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:21.601703 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.601669    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-podinfo" pod "8cbf6236-95c3-4847-9650-f5de5af07ad2" (UID: "8cbf6236-95c3-4847-9650-f5de5af07ad2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:54:21.601802 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.601738    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-token" pod "8cbf6236-95c3-4847-9650-f5de5af07ad2" (UID: "8cbf6236-95c3-4847-9650-f5de5af07ad2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:54:21.601971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.601936    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-envoy" pod "8cbf6236-95c3-4847-9650-f5de5af07ad2" (UID: "8cbf6236-95c3-4847-9650-f5de5af07ad2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:21.601971 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.601954    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-kube-api-access-jzwtk" pod "8cbf6236-95c3-4847-9650-f5de5af07ad2" (UID: "8cbf6236-95c3-4847-9650-f5de5af07ad2"). InnerVolumeSpecName "kube-api-access-jzwtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:54:21.701459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.701431    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:21.701459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.701455    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzwtk\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-kube-api-access-jzwtk\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:21.701636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.701467    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:21.701636 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:21.701475    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbf6236-95c3-4847-9650-f5de5af07ad2-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:22.419094 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:22.419063    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"
Nov 02 21:54:22.419552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:22.419128    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv" event={"ID":"8cbf6236-95c3-4847-9650-f5de5af07ad2","Type":"ContainerDied","Data":"21df70c30fbc5ab2f3c63ef473ee556e55d515409a8c4ec2df867a0274ae67ba"}
Nov 02 21:54:22.419552 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:22.419186    4528 scope.go:122] "RemoveContainer" containerID="25171310f502d4f2cf575d94f65652e88b6c7233930515e9dea49c02ebebb601"
Nov 02 21:54:22.613803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:22.613774    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"]
Nov 02 21:54:22.663860 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:22.663832    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-eabda2-openshift-default-df6b8ff7-xrxfv"]
Nov 02 21:54:22.914007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:22.913976    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" path="/var/lib/kubelet/pods/8cbf6236-95c3-4847-9650-f5de5af07ad2/volumes"
Nov 02 21:54:23.991091 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:23.991055    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"]
Nov 02 21:54:24.097549 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:24.097518    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"]
Nov 02 21:54:28.992151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:28.992098    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5" podUID="2c85644c-b5a3-4845-9f93-6e1afbeb66ca" containerName="payload-processing" containerID="cri-o://262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96" gracePeriod=25
Nov 02 21:54:29.098839 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.098802    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv" podUID="fee16119-ee33-4dd9-9764-465ad0c3c241" containerName="payload-pre-processing" containerID="cri-o://d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3" gracePeriod=25
Nov 02 21:54:29.284102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.284073    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:54:29.323669 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.323627    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-plugins-config-volume\") pod \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\" (UID: \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\") "
Nov 02 21:54:29.323851 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.323730    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-kube-api-access-tz7xm\") pod \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\" (UID: \"2c85644c-b5a3-4845-9f93-6e1afbeb66ca\") "
Nov 02 21:54:29.324025 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.324001    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-plugins-config-volume" pod "2c85644c-b5a3-4845-9f93-6e1afbeb66ca" (UID: "2c85644c-b5a3-4845-9f93-6e1afbeb66ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:54:29.324314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.324297    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:29.326076 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.326044    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-kube-api-access-tz7xm" pod "2c85644c-b5a3-4845-9f93-6e1afbeb66ca" (UID: "2c85644c-b5a3-4845-9f93-6e1afbeb66ca"). InnerVolumeSpecName "kube-api-access-tz7xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:54:29.345224 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.345206    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:54:29.425187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.425152    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee16119-ee33-4dd9-9764-465ad0c3c241-kube-api-access-9pjk7\" (UniqueName: \"kubernetes.io/projected/fee16119-ee33-4dd9-9764-465ad0c3c241-kube-api-access-9pjk7\") pod \"fee16119-ee33-4dd9-9764-465ad0c3c241\" (UID: \"fee16119-ee33-4dd9-9764-465ad0c3c241\") "
Nov 02 21:54:29.425427 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.425251    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee16119-ee33-4dd9-9764-465ad0c3c241-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee16119-ee33-4dd9-9764-465ad0c3c241-plugins-config-volume\") pod \"fee16119-ee33-4dd9-9764-465ad0c3c241\" (UID: \"fee16119-ee33-4dd9-9764-465ad0c3c241\") "
Nov 02 21:54:29.426125 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.426073    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee16119-ee33-4dd9-9764-465ad0c3c241-plugins-config-volume" pod "fee16119-ee33-4dd9-9764-465ad0c3c241" (UID: "fee16119-ee33-4dd9-9764-465ad0c3c241"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:54:29.426728 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.426700    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/2c85644c-b5a3-4845-9f93-6e1afbeb66ca-kube-api-access-tz7xm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:29.426881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.426863    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee16119-ee33-4dd9-9764-465ad0c3c241-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:29.434616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.434587    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee16119-ee33-4dd9-9764-465ad0c3c241-kube-api-access-9pjk7" pod "fee16119-ee33-4dd9-9764-465ad0c3c241" (UID: "fee16119-ee33-4dd9-9764-465ad0c3c241"). InnerVolumeSpecName "kube-api-access-9pjk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:54:29.599924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.588237    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pjk7\" (UniqueName: \"kubernetes.io/projected/fee16119-ee33-4dd9-9764-465ad0c3c241-kube-api-access-9pjk7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:29.632006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.631952    4528 generic.go:356] "Generic (PLEG): container finished" podID="fee16119-ee33-4dd9-9764-465ad0c3c241" containerID="d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3" exitCode=0
Nov 02 21:54:29.632307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.632137    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv" event={"ID":"fee16119-ee33-4dd9-9764-465ad0c3c241","Type":"ContainerDied","Data":"d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3"}
Nov 02 21:54:29.632307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.632211    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv" event={"ID":"fee16119-ee33-4dd9-9764-465ad0c3c241","Type":"ContainerDied","Data":"3b9c689a6893388df176d8293597ff8c71c8774dacaeb9ef1d5d2d056f3c4fe6"}
Nov 02 21:54:29.632307 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.632239    4528 scope.go:122] "RemoveContainer" containerID="d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3"
Nov 02 21:54:29.632603 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.632385    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"
Nov 02 21:54:29.644256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.644228    4528 generic.go:356] "Generic (PLEG): container finished" podID="2c85644c-b5a3-4845-9f93-6e1afbeb66ca" containerID="262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96" exitCode=0
Nov 02 21:54:29.644591 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.644571    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5" event={"ID":"2c85644c-b5a3-4845-9f93-6e1afbeb66ca","Type":"ContainerDied","Data":"262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96"}
Nov 02 21:54:29.644777 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.644759    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5" event={"ID":"2c85644c-b5a3-4845-9f93-6e1afbeb66ca","Type":"ContainerDied","Data":"8d21d3f64502d09d2be01b473a13baf9185c352b85dd967e34d29bd30d0bf64b"}
Nov 02 21:54:29.645010 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.644995    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"
Nov 02 21:54:29.770219 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.769867    4528 scope.go:122] "RemoveContainer" containerID="d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3"
Nov 02 21:54:29.770808 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:54:29.770461    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3\": container with ID starting with d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3 not found: ID does not exist" containerID="d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3"
Nov 02 21:54:29.770808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.770499    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3"} err="failed to get container status \"d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3\": rpc error: code = NotFound desc = could not find container \"d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3\": container with ID starting with d520dc972e9ebf2effda617d5a0008e07eb08cefb13dd96df4bb754805633ee3 not found: ID does not exist"
Nov 02 21:54:29.770808 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.770521    4528 scope.go:122] "RemoveContainer" containerID="262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96"
Nov 02 21:54:29.790794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.790759    4528 scope.go:122] "RemoveContainer" containerID="262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96"
Nov 02 21:54:29.791453 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:54:29.791422    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96\": container with ID starting with 262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96 not found: ID does not exist" containerID="262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96"
Nov 02 21:54:29.791768 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.791464    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96"} err="failed to get container status \"262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96\": rpc error: code = NotFound desc = could not find container \"262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96\": container with ID starting with 262042b40b66a21aea0359f821cf2ae41c19f70ad9bcb3380b956af10ccc8d96 not found: ID does not exist"
Nov 02 21:54:29.910187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.907989    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"]
Nov 02 21:54:29.935217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.935160    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3c7706-8585f7bc9-4jsqv"]
Nov 02 21:54:29.969699 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.969469    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"]
Nov 02 21:54:29.975099 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:29.975073    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3c7706-db58556b9-scsn5"]
Nov 02 21:54:30.999127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:30.997985    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c85644c-b5a3-4845-9f93-6e1afbeb66ca" path="/var/lib/kubelet/pods/2c85644c-b5a3-4845-9f93-6e1afbeb66ca/volumes"
Nov 02 21:54:30.999127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:30.998603    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee16119-ee33-4dd9-9764-465ad0c3c241" path="/var/lib/kubelet/pods/fee16119-ee33-4dd9-9764-465ad0c3c241/volumes"
Nov 02 21:54:39.049324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:39.049285    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"]
Nov 02 21:54:39.099717 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:39.099687    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0701606c-bbk2g"]
Nov 02 21:54:39.781401 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:39.781367    4528 generic.go:356] "Generic (PLEG): container finished" podID="b33afd9e-0e68-4231-910d-28348af5ab93" containerID="c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b" exitCode=0
Nov 02 21:54:39.781603 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:39.781441    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" event={"ID":"b33afd9e-0e68-4231-910d-28348af5ab93","Type":"ContainerDied","Data":"c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b"}
Nov 02 21:54:39.781913 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:39.781897    4528 scope.go:122] "RemoveContainer" containerID="c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b"
Nov 02 21:54:39.783380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:39.783352    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:54:39.783456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:39.783420    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:54:39.783456 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:39.783452    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:54:40.793279 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:40.793242    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" event={"ID":"b33afd9e-0e68-4231-910d-28348af5ab93","Type":"ContainerStarted","Data":"e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5"}
Nov 02 21:54:40.934166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:40.934128    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59da25a0-08cf-4900-9c8e-c124c839a922" path="/var/lib/kubelet/pods/59da25a0-08cf-4900-9c8e-c124c839a922/volumes"
Nov 02 21:54:41.426134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:41.424374    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:54:41.426134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:41.424487    4528 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 02 21:54:41.426134 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:41.424528    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 02 21:54:42.431540 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:42.431499    4528 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 02 21:54:42.432209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:42.432176    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 02 21:54:43.426434 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:43.426400    4528 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 02 21:54:43.426648 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:43.426489    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 02 21:54:44.134917 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:54:44.134874    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:54:44.433019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:44.432489    4528 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 02 21:54:44.433019 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:44.432548    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 02 21:54:45.426493 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:45.426451    4528 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 02 21:54:45.427082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:45.426525    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 02 21:54:46.425894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:46.425866    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:54:46.426351 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:46.426330    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:54:46.426920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:46.426901    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 21:54:46.991611 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:46.991577    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"]
Nov 02 21:54:46.991890 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:46.991859    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy" containerID="cri-o://77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18" gracePeriod=30
Nov 02 21:54:47.287997 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:47.287966    4528 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 21:54:47.288206 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:47.288019    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 21:54:52.257616 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.257562    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:54:52.346317 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346293    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-certs\") pod \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") "
Nov 02 21:54:52.346468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346322    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-envoy\") pod \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") "
Nov 02 21:54:52.346468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346378    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-data\") pod \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") "
Nov 02 21:54:52.346468 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346451    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-socket\") pod \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") "
Nov 02 21:54:52.346620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346513    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-podinfo\") pod \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") "
Nov 02 21:54:52.346620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346539    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istiod-ca-cert\") pod \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") "
Nov 02 21:54:52.346620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346546    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-certs" pod "0ebad026-b9bc-4ca1-8626-79ec06333aeb" (UID: "0ebad026-b9bc-4ca1-8626-79ec06333aeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:52.346620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346573    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-token\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-token\") pod \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") "
Nov 02 21:54:52.346835 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346748    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-kube-api-access-rpg6w\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-kube-api-access-rpg6w\") pod \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") "
Nov 02 21:54:52.346835 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346769    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-socket" pod "0ebad026-b9bc-4ca1-8626-79ec06333aeb" (UID: "0ebad026-b9bc-4ca1-8626-79ec06333aeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:52.346835 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346778    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-data" pod "0ebad026-b9bc-4ca1-8626-79ec06333aeb" (UID: "0ebad026-b9bc-4ca1-8626-79ec06333aeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:52.346835 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.346822    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-credential-socket\") pod \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\" (UID: \"0ebad026-b9bc-4ca1-8626-79ec06333aeb\") "
Nov 02 21:54:52.347125 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.347069    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-credential-socket" pod "0ebad026-b9bc-4ca1-8626-79ec06333aeb" (UID: "0ebad026-b9bc-4ca1-8626-79ec06333aeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:52.347256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.347226    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istiod-ca-cert" pod "0ebad026-b9bc-4ca1-8626-79ec06333aeb" (UID: "0ebad026-b9bc-4ca1-8626-79ec06333aeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:54:52.347481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.347449    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:52.347481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.347475    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:52.347631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.347490    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:52.347631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.347536    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:52.347631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.347573    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:52.348771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.348748    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-envoy" pod "0ebad026-b9bc-4ca1-8626-79ec06333aeb" (UID: "0ebad026-b9bc-4ca1-8626-79ec06333aeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:54:52.348771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.348761    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-podinfo" pod "0ebad026-b9bc-4ca1-8626-79ec06333aeb" (UID: "0ebad026-b9bc-4ca1-8626-79ec06333aeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:54:52.348896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.348761    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-token" pod "0ebad026-b9bc-4ca1-8626-79ec06333aeb" (UID: "0ebad026-b9bc-4ca1-8626-79ec06333aeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:54:52.349021 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.348998    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-kube-api-access-rpg6w" pod "0ebad026-b9bc-4ca1-8626-79ec06333aeb" (UID: "0ebad026-b9bc-4ca1-8626-79ec06333aeb"). InnerVolumeSpecName "kube-api-access-rpg6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:54:52.449161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.449104    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:52.449161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.449157    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:52.449161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.449167    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:52.449402 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.449176    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpg6w\" (UniqueName: \"kubernetes.io/projected/0ebad026-b9bc-4ca1-8626-79ec06333aeb-kube-api-access-rpg6w\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:54:52.897874 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.897837    4528 generic.go:356] "Generic (PLEG): container finished" podID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerID="77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18" exitCode=0
Nov 02 21:54:52.898090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.897888    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"
Nov 02 21:54:52.898090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.897924    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" event={"ID":"0ebad026-b9bc-4ca1-8626-79ec06333aeb","Type":"ContainerDied","Data":"77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18"}
Nov 02 21:54:52.898090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.898014    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4" event={"ID":"0ebad026-b9bc-4ca1-8626-79ec06333aeb","Type":"ContainerDied","Data":"e1a97722071578f72ee4c40254745aa5138aac7cfb931f0f93ab99424d4bb60c"}
Nov 02 21:54:52.898090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.898047    4528 scope.go:122] "RemoveContainer" containerID="77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18"
Nov 02 21:54:52.908945 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.908931    4528 scope.go:122] "RemoveContainer" containerID="154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82"
Nov 02 21:54:52.917920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.917906    4528 scope.go:122] "RemoveContainer" containerID="77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18"
Nov 02 21:54:52.918157 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:54:52.918141    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18\": container with ID starting with 77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18 not found: ID does not exist" containerID="77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18"
Nov 02 21:54:52.918223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.918164    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18"} err="failed to get container status \"77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18\": rpc error: code = NotFound desc = could not find container \"77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18\": container with ID starting with 77742601e682fa4eac8e8ceeecb433efc2f793f62210cd52d8c28540f1a93a18 not found: ID does not exist"
Nov 02 21:54:52.918223 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.918180    4528 scope.go:122] "RemoveContainer" containerID="154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82"
Nov 02 21:54:52.918392 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:54:52.918373    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82\": container with ID starting with 154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82 not found: ID does not exist" containerID="154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82"
Nov 02 21:54:52.918449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:52.918400    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82"} err="failed to get container status \"154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82\": rpc error: code = NotFound desc = could not find container \"154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82\": container with ID starting with 154e8dd43126a66d24a711eef9c7b34d59aa0042818b2fb6c5dd4c075e034e82 not found: ID does not exist"
Nov 02 21:54:53.067785 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:53.067755    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"]
Nov 02 21:54:53.117686 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:53.117657    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-5fnp4"]
Nov 02 21:54:53.651171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:53.651142    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:54:54.914927 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.914896    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" path="/var/lib/kubelet/pods/0ebad026-b9bc-4ca1-8626-79ec06333aeb/volumes"
Nov 02 21:54:54.943289 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.943255    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"]
Nov 02 21:54:54.943962 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.943946    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5834a1eb-f3fc-45a9-a974-4717a6b46b8f" containerName="payload-processing"
Nov 02 21:54:54.944033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.943964    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5834a1eb-f3fc-45a9-a974-4717a6b46b8f" containerName="payload-processing"
Nov 02 21:54:54.944033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.943999    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy"
Nov 02 21:54:54.944033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944008    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy"
Nov 02 21:54:54.944033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944022    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fee16119-ee33-4dd9-9764-465ad0c3c241" containerName="payload-pre-processing"
Nov 02 21:54:54.944033 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944029    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee16119-ee33-4dd9-9764-465ad0c3c241" containerName="payload-pre-processing"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944047    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0f992866-c294-4889-a086-8b00a6fae157" containerName="revoke-keys"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944055    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f992866-c294-4889-a086-8b00a6fae157" containerName="revoke-keys"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944078    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2c85644c-b5a3-4845-9f93-6e1afbeb66ca" containerName="payload-processing"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944085    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c85644c-b5a3-4845-9f93-6e1afbeb66ca" containerName="payload-processing"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944100    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944121    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944141    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944147    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944158    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy"
Nov 02 21:54:54.944230 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944164    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy"
Nov 02 21:54:54.944588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944272    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5834a1eb-f3fc-45a9-a974-4717a6b46b8f" containerName="payload-processing"
Nov 02 21:54:54.944588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944288    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy"
Nov 02 21:54:54.944588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944297    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee16119-ee33-4dd9-9764-465ad0c3c241" containerName="payload-pre-processing"
Nov 02 21:54:54.944588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944307    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f992866-c294-4889-a086-8b00a6fae157" containerName="revoke-keys"
Nov 02 21:54:54.944588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944318    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c85644c-b5a3-4845-9f93-6e1afbeb66ca" containerName="payload-processing"
Nov 02 21:54:54.944588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944326    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebad026-b9bc-4ca1-8626-79ec06333aeb" containerName="istio-proxy"
Nov 02 21:54:54.944588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944337    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy"
Nov 02 21:54:54.944588 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.944346    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbf6236-95c3-4847-9650-f5de5af07ad2" containerName="istio-proxy"
Nov 02 21:54:54.949204 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:54.949184    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.007236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.007206    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"]
Nov 02 21:54:55.051196 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.051172    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-a1d751-openshift-default-dockercfg-57whc\""
Nov 02 21:54:55.155244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.155205    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-socket\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.155610 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.155573    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-data\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.155876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.155859    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-token\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.156090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.156075    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-podinfo\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.156296 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.156281    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-envoy\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.156506 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.156484    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-credential-socket\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.163132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.156694    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt48b\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-kube-api-access-nt48b\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.163445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.163423    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istiod-ca-cert\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.163677 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.163658    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-certs\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.264902    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt48b\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-kube-api-access-nt48b\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.264997    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istiod-ca-cert\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.265068    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-certs\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.265105    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-socket\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.265151    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-data\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.265197    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-token\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.265289    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-podinfo\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.265330    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-envoy\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265687 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.265393    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-credential-socket\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.265924 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.265836    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-credential-socket\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.266847 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.266810    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istiod-ca-cert\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.268186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.267129    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-certs\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.268186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.267416    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-socket\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.268186 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.267668    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-data\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.283827 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.283757    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-podinfo\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.284977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.284927    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-envoy\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.323290 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.323260    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt48b\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-kube-api-access-nt48b\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.324239 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.324214    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-token\") pod \"e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") " pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.562199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.562103    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:55.980882 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:54:55.980847    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adfc106_9a49_44d2_b5c7_cca45733f6bd.slice/crio-40a57aa0927c54e6d051d1c413e5b528208660865019ba10601278ba7cf96fd1 WatchSource:0}: Error finding container 40a57aa0927c54e6d051d1c413e5b528208660865019ba10601278ba7cf96fd1: Status 404 returned error can't find the container with id 40a57aa0927c54e6d051d1c413e5b528208660865019ba10601278ba7cf96fd1
Nov 02 21:54:55.983656 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.983625    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:54:55.983751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.983690    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:54:55.983751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.983735    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:54:55.984587 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:55.984566    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"]
Nov 02 21:54:56.926982 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:56.926950    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" event={"ID":"1adfc106-9a49-44d2-b5c7-cca45733f6bd","Type":"ContainerStarted","Data":"123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a"}
Nov 02 21:54:56.927168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:56.926993    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" event={"ID":"1adfc106-9a49-44d2-b5c7-cca45733f6bd","Type":"ContainerStarted","Data":"40a57aa0927c54e6d051d1c413e5b528208660865019ba10601278ba7cf96fd1"}
Nov 02 21:54:57.048357 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:57.048327    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"]
Nov 02 21:54:57.051530 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:57.051511    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e1b4aa-g7sfs"]
Nov 02 21:54:57.095371 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:57.095321    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" podStartSLOduration=3.095308784 podStartE2EDuration="3.095308784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:57.095050856 +0000 UTC m=+2784.358587172" watchObservedRunningTime="2026-11-02 21:54:57.095308784 +0000 UTC m=+2784.358845100"
Nov 02 21:54:57.562953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:57.562914    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:57.569042 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:57.569008    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:57.933996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:57.933918    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:57.934840 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:57.934813    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:54:58.912882 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:54:58.912844    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a2d9c9-eba9-48bb-bc4a-4519424265ad" path="/var/lib/kubelet/pods/42a2d9c9-eba9-48bb-bc4a-4519424265ad/volumes"
Nov 02 21:55:06.118151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.118094    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"]
Nov 02 21:55:06.123030 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.123008    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.179853 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.179817    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-0f0874-openshift-default-dockercfg-jzjt8\""
Nov 02 21:55:06.206283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.206244    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-data\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.231138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.217289    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hvnd\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-kube-api-access-4hvnd\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.231138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.217359    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-certs\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.231138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.217385    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387be747-622a-4118-89c1-9565f9c5e947-istiod-ca-cert\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.231138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.217436    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387be747-622a-4118-89c1-9565f9c5e947-istio-podinfo\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.231138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.217488    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-credential-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.231138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.217528    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-istio-token\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.231138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.217558    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.231138 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.217612    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-envoy\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.293161 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.293104    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"]
Nov 02 21:55:06.318948 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.318912    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hvnd\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-kube-api-access-4hvnd\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.318979    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-certs\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.319009    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387be747-622a-4118-89c1-9565f9c5e947-istiod-ca-cert\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.319054    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387be747-622a-4118-89c1-9565f9c5e947-istio-podinfo\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.319100    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-credential-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.319157    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-istio-token\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319199 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.319189    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.319245    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-envoy\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319734 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.319305    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-data\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.319758    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-credential-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.319930 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.319913    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-data\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.320298 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.320275    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.320532 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.320509    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-certs\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.321175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.321149    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387be747-622a-4118-89c1-9565f9c5e947-istiod-ca-cert\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.322509 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.322487    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-envoy\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.394758 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.383642    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387be747-622a-4118-89c1-9565f9c5e947-istio-podinfo\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.395623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.395593    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-istio-token\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.424131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.406904    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hvnd\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-kube-api-access-4hvnd\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:06.458158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:06.456389    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:07.016102 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.016060    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:55:07.016236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.016176    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:55:07.016236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.016220    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:55:07.034847 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.031646    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"]
Nov 02 21:55:07.040736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.040701    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" event={"ID":"387be747-622a-4118-89c1-9565f9c5e947","Type":"ContainerStarted","Data":"ae8b0970a7701a36db8d8b0973262580cefcf92e4eabc90ba6a4f9d62847814d"}
Nov 02 21:55:07.047385 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.047361    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"]
Nov 02 21:55:07.052846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.052827    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:07.104521 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.104494    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"]
Nov 02 21:55:07.105428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.105410    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a1d751\""
Nov 02 21:55:07.105660 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.105642    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a1d751-dockercfg-hkzzt\""
Nov 02 21:55:07.134684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.134640    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a2a5be-4801-4fc9-bb1f-5804be821e95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw\" (UID: \"53a2a5be-4801-4fc9-bb1f-5804be821e95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:07.135175 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.134867    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrcx5\" (UniqueName: \"kubernetes.io/projected/53a2a5be-4801-4fc9-bb1f-5804be821e95-kube-api-access-mrcx5\") pod \"payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw\" (UID: \"53a2a5be-4801-4fc9-bb1f-5804be821e95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:07.253822 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.253779    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"]
Nov 02 21:55:07.299220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.263140    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrcx5\" (UniqueName: \"kubernetes.io/projected/53a2a5be-4801-4fc9-bb1f-5804be821e95-kube-api-access-mrcx5\") pod \"payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw\" (UID: \"53a2a5be-4801-4fc9-bb1f-5804be821e95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:07.299220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.263328    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a2a5be-4801-4fc9-bb1f-5804be821e95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw\" (UID: \"53a2a5be-4801-4fc9-bb1f-5804be821e95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:07.299220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.264371    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a2a5be-4801-4fc9-bb1f-5804be821e95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw\" (UID: \"53a2a5be-4801-4fc9-bb1f-5804be821e95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:07.299220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.284429    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:07.299220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.297627    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"]
Nov 02 21:55:07.368495 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.367267    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl\" (UID: \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:07.368495 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.367396    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmvx\" (UniqueName: \"kubernetes.io/projected/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-kube-api-access-bhmvx\") pod \"payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl\" (UID: \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:07.401151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.374666    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrcx5\" (UniqueName: \"kubernetes.io/projected/53a2a5be-4801-4fc9-bb1f-5804be821e95-kube-api-access-mrcx5\") pod \"payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw\" (UID: \"53a2a5be-4801-4fc9-bb1f-5804be821e95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:07.469336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.469304    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl\" (UID: \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:07.469514 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.469453    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmvx\" (UniqueName: \"kubernetes.io/projected/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-kube-api-access-bhmvx\") pod \"payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl\" (UID: \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:07.470083 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.470063    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl\" (UID: \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:07.526238 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.526209    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmvx\" (UniqueName: \"kubernetes.io/projected/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-kube-api-access-bhmvx\") pod \"payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl\" (UID: \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:07.673593 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.673512    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:07.674573 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:07.674554    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:08.047510 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:08.047481    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a7ebec_5acc_422b_b4e5_d6a0d6a4ace6.slice/crio-4d301e51f47be4f8f520201f9bd97e2aa73428970283f706b9830ac9f3ed0655 WatchSource:0}: Error finding container 4d301e51f47be4f8f520201f9bd97e2aa73428970283f706b9830ac9f3ed0655: Status 404 returned error can't find the container with id 4d301e51f47be4f8f520201f9bd97e2aa73428970283f706b9830ac9f3ed0655
Nov 02 21:55:08.050945 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:08.050923    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"]
Nov 02 21:55:08.054160 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:08.054136    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" event={"ID":"387be747-622a-4118-89c1-9565f9c5e947","Type":"ContainerStarted","Data":"37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1"}
Nov 02 21:55:08.092290 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:08.092268    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a2a5be_4801_4fc9_bb1f_5804be821e95.slice/crio-907f926025659fb34b070d28dc0bd0bcd2e869e2c46956a14dd6f8c398dee1dc WatchSource:0}: Error finding container 907f926025659fb34b070d28dc0bd0bcd2e869e2c46956a14dd6f8c398dee1dc: Status 404 returned error can't find the container with id 907f926025659fb34b070d28dc0bd0bcd2e869e2c46956a14dd6f8c398dee1dc
Nov 02 21:55:08.096892 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:08.096864    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"]
Nov 02 21:55:08.175104 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:08.175056    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" podStartSLOduration=2.17504293 podStartE2EDuration="2.17504293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:08.17077427 +0000 UTC m=+2795.434310586" watchObservedRunningTime="2026-11-02 21:55:08.17504293 +0000 UTC m=+2795.438579246"
Nov 02 21:55:08.457084 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:08.457002    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:08.462620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:08.462594    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:09.062058 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.062017    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl" event={"ID":"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6","Type":"ContainerStarted","Data":"9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e"}
Nov 02 21:55:09.062240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.062088    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:09.062240 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.062129    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl" event={"ID":"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6","Type":"ContainerStarted","Data":"4d301e51f47be4f8f520201f9bd97e2aa73428970283f706b9830ac9f3ed0655"}
Nov 02 21:55:09.063624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.063592    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw" event={"ID":"53a2a5be-4801-4fc9-bb1f-5804be821e95","Type":"ContainerStarted","Data":"1d9ea3ae6e26992e090dff65bcef2030dd8df7d62059d474aa3a287cad2e5cf3"}
Nov 02 21:55:09.063761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.063637    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw" event={"ID":"53a2a5be-4801-4fc9-bb1f-5804be821e95","Type":"ContainerStarted","Data":"907f926025659fb34b070d28dc0bd0bcd2e869e2c46956a14dd6f8c398dee1dc"}
Nov 02 21:55:09.064090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.064063    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:09.064222 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.064129    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:09.064776 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.064762    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:55:09.178584 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.178536    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl" podStartSLOduration=2.178523108 podStartE2EDuration="2.178523108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:09.175450067 +0000 UTC m=+2796.438986384" watchObservedRunningTime="2026-11-02 21:55:09.178523108 +0000 UTC m=+2796.442059449"
Nov 02 21:55:09.412299 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:09.412186    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw" podStartSLOduration=3.412170541 podStartE2EDuration="3.412170541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:09.410318623 +0000 UTC m=+2796.673854961" watchObservedRunningTime="2026-11-02 21:55:09.412170541 +0000 UTC m=+2796.675706858"
Nov 02 21:55:13.051507 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:13.051466    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"]
Nov 02 21:55:13.102325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:13.102296    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4c8948-t54sz"]
Nov 02 21:55:14.113968 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:55:14.113936    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:55:14.522211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.522177    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"]
Nov 02 21:55:14.526876 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.526856    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:14.579976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.579953    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"]
Nov 02 21:55:14.612174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.612141    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c090bef4-b276-45ba-9754-834012be133f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl\" (UID: \"c090bef4-b276-45ba-9754-834012be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:14.612350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.612236    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5972j\" (UniqueName: \"kubernetes.io/projected/c090bef4-b276-45ba-9754-834012be133f-kube-api-access-5972j\") pod \"payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl\" (UID: \"c090bef4-b276-45ba-9754-834012be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:14.649132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.628388    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-0f0874\""
Nov 02 21:55:14.649132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.628510    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-0f0874-dockercfg-kg8rm\""
Nov 02 21:55:14.710130 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.705174    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"]
Nov 02 21:55:14.733260 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.716902    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c090bef4-b276-45ba-9754-834012be133f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl\" (UID: \"c090bef4-b276-45ba-9754-834012be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:14.733260 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.717019    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5972j\" (UniqueName: \"kubernetes.io/projected/c090bef4-b276-45ba-9754-834012be133f-kube-api-access-5972j\") pod \"payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl\" (UID: \"c090bef4-b276-45ba-9754-834012be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:14.733260 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.718455    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c090bef4-b276-45ba-9754-834012be133f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl\" (UID: \"c090bef4-b276-45ba-9754-834012be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:14.733260 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.729621    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:14.761429 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.761398    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"]
Nov 02 21:55:14.774906 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.774829    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5972j\" (UniqueName: \"kubernetes.io/projected/c090bef4-b276-45ba-9754-834012be133f-kube-api-access-5972j\") pod \"payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl\" (UID: \"c090bef4-b276-45ba-9754-834012be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:14.840469 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.840437    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:14.917271 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.917241    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e473c7b6-57d8-4610-a0f0-87d759da005e" path="/var/lib/kubelet/pods/e473c7b6-57d8-4610-a0f0-87d759da005e/volumes"
Nov 02 21:55:14.920819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.920795    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5\" (UID: \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:14.920943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:14.920835    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxsl8\" (UniqueName: \"kubernetes.io/projected/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-kube-api-access-gxsl8\") pod \"payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5\" (UID: \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:15.021881 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:15.021849    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5\" (UID: \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:15.022031 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:15.021891    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxsl8\" (UniqueName: \"kubernetes.io/projected/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-kube-api-access-gxsl8\") pod \"payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5\" (UID: \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:15.022516 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:15.022497    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5\" (UID: \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:15.079359 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:15.079296    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxsl8\" (UniqueName: \"kubernetes.io/projected/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-kube-api-access-gxsl8\") pod \"payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5\" (UID: \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:15.209224 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:15.209193    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc090bef4_b276_45ba_9754_834012be133f.slice/crio-245204a3f1002bc4a8c810084058e94d06cc1fba56c920748498006b31ff0eab WatchSource:0}: Error finding container 245204a3f1002bc4a8c810084058e94d06cc1fba56c920748498006b31ff0eab: Status 404 returned error can't find the container with id 245204a3f1002bc4a8c810084058e94d06cc1fba56c920748498006b31ff0eab
Nov 02 21:55:15.214078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:15.214055    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"]
Nov 02 21:55:15.367197 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:15.366337    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:15.744516 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:15.744483    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bd8bfc_643e_4750_99c2_d6bbcfa6abf5.slice/crio-e0be525b83ea1f7a67259c209ada420d75c01179f81efe2f8d5a1c2747bf7a70 WatchSource:0}: Error finding container e0be525b83ea1f7a67259c209ada420d75c01179f81efe2f8d5a1c2747bf7a70: Status 404 returned error can't find the container with id e0be525b83ea1f7a67259c209ada420d75c01179f81efe2f8d5a1c2747bf7a70
Nov 02 21:55:15.747562 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:15.747539    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"]
Nov 02 21:55:16.112846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:16.112378    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5" event={"ID":"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5","Type":"ContainerStarted","Data":"47930b1d0f847ab72afb0ea72d0d35695d5141c76c31118fcdddf45cfe231a17"}
Nov 02 21:55:16.112846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:16.112448    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:16.112846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:16.112468    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5" event={"ID":"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5","Type":"ContainerStarted","Data":"e0be525b83ea1f7a67259c209ada420d75c01179f81efe2f8d5a1c2747bf7a70"}
Nov 02 21:55:16.114169 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:16.114139    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl" event={"ID":"c090bef4-b276-45ba-9754-834012be133f","Type":"ContainerStarted","Data":"3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c"}
Nov 02 21:55:16.114169 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:16.114186    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl" event={"ID":"c090bef4-b276-45ba-9754-834012be133f","Type":"ContainerStarted","Data":"245204a3f1002bc4a8c810084058e94d06cc1fba56c920748498006b31ff0eab"}
Nov 02 21:55:16.114417 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:16.114228    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:16.229748 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:16.229702    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5" podStartSLOduration=2.229688562 podStartE2EDuration="2.229688562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:16.225504082 +0000 UTC m=+2803.489040400" watchObservedRunningTime="2026-11-02 21:55:16.229688562 +0000 UTC m=+2803.493224886"
Nov 02 21:55:16.341394 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:16.341343    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl" podStartSLOduration=2.341328831 podStartE2EDuration="2.341328831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:16.338769403 +0000 UTC m=+2803.602305721" watchObservedRunningTime="2026-11-02 21:55:16.341328831 +0000 UTC m=+2803.604865144"
Nov 02 21:55:17.102556 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.102523    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"]
Nov 02 21:55:17.106922 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.106905    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.159211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.159184    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 21:55:17.159367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.159325    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"]
Nov 02 21:55:17.171308 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.171279    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f619a351-bmnlx\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.171623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.171337    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f619a351-bmnlx\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.171623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.171386    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8994\" (UniqueName: \"kubernetes.io/projected/94bd94f6-eeb7-496c-9a76-68ce01b9275a-kube-api-access-v8994\") pod \"maas-api-revoke-sub-f619a351-bmnlx\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.205529 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.205506    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:55:17.205647 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.205550    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 21:55:17.209174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.209158    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 21:55:17.272173 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.272141    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f619a351-bmnlx\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.272567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.272189    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8994\" (UniqueName: \"kubernetes.io/projected/94bd94f6-eeb7-496c-9a76-68ce01b9275a-kube-api-access-v8994\") pod \"maas-api-revoke-sub-f619a351-bmnlx\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.272567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.272441    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f619a351-bmnlx\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.272800 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.272784    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f619a351-bmnlx\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.274694 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.274670    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f619a351-bmnlx\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.329284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.329261    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8994\" (UniqueName: \"kubernetes.io/projected/94bd94f6-eeb7-496c-9a76-68ce01b9275a-kube-api-access-v8994\") pod \"maas-api-revoke-sub-f619a351-bmnlx\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.420450 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.420385    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:17.786226 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:17.786197    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bd94f6_eeb7_496c_9a76_68ce01b9275a.slice/crio-602055f28ecd7a2a90a021b79bc0436255632bde50b08d32479c57728a62c687 WatchSource:0}: Error finding container 602055f28ecd7a2a90a021b79bc0436255632bde50b08d32479c57728a62c687: Status 404 returned error can't find the container with id 602055f28ecd7a2a90a021b79bc0436255632bde50b08d32479c57728a62c687
Nov 02 21:55:17.791759 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:17.791734    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"]
Nov 02 21:55:18.129937 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:18.129907    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx" event={"ID":"94bd94f6-eeb7-496c-9a76-68ce01b9275a","Type":"ContainerStarted","Data":"e92efdd5b620dfc967df6dda55fd89735df6d8649c3965ab02d9fbd662fe2d45"}
Nov 02 21:55:18.130075 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:18.129959    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx" event={"ID":"94bd94f6-eeb7-496c-9a76-68ce01b9275a","Type":"ContainerStarted","Data":"602055f28ecd7a2a90a021b79bc0436255632bde50b08d32479c57728a62c687"}
Nov 02 21:55:18.248363 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:18.248311    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx" podStartSLOduration=1.248297766 podStartE2EDuration="1.248297766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:18.244766126 +0000 UTC m=+2805.508302442" watchObservedRunningTime="2026-11-02 21:55:18.248297766 +0000 UTC m=+2805.511834082"
Nov 02 21:55:19.137827 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:19.137794    4528 generic.go:356] "Generic (PLEG): container finished" podID="94bd94f6-eeb7-496c-9a76-68ce01b9275a" containerID="e92efdd5b620dfc967df6dda55fd89735df6d8649c3965ab02d9fbd662fe2d45" exitCode=0
Nov 02 21:55:19.138309 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:19.137880    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx" event={"ID":"94bd94f6-eeb7-496c-9a76-68ce01b9275a","Type":"ContainerDied","Data":"e92efdd5b620dfc967df6dda55fd89735df6d8649c3965ab02d9fbd662fe2d45"}
Nov 02 21:55:20.073037 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.073011    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 21:55:20.326511 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.326458    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:20.410843 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.410815    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-openshift-ca\") pod \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") "
Nov 02 21:55:20.411012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.410867    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bd94f6-eeb7-496c-9a76-68ce01b9275a-kube-api-access-v8994\" (UniqueName: \"kubernetes.io/projected/94bd94f6-eeb7-496c-9a76-68ce01b9275a-kube-api-access-v8994\") pod \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") "
Nov 02 21:55:20.411012 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.410936    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-xks-ca\") pod \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\" (UID: \"94bd94f6-eeb7-496c-9a76-68ce01b9275a\") "
Nov 02 21:55:20.411242 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.411213    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-openshift-ca" pod "94bd94f6-eeb7-496c-9a76-68ce01b9275a" (UID: "94bd94f6-eeb7-496c-9a76-68ce01b9275a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:55:20.411328 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.411296    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:20.413082 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.413061    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-xks-ca" pod "94bd94f6-eeb7-496c-9a76-68ce01b9275a" (UID: "94bd94f6-eeb7-496c-9a76-68ce01b9275a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:55:20.413288 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.413270    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bd94f6-eeb7-496c-9a76-68ce01b9275a-kube-api-access-v8994" pod "94bd94f6-eeb7-496c-9a76-68ce01b9275a" (UID: "94bd94f6-eeb7-496c-9a76-68ce01b9275a"). InnerVolumeSpecName "kube-api-access-v8994". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:55:20.512430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.512395    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8994\" (UniqueName: \"kubernetes.io/projected/94bd94f6-eeb7-496c-9a76-68ce01b9275a-kube-api-access-v8994\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:20.512430 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:20.512428    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94bd94f6-eeb7-496c-9a76-68ce01b9275a-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:21.079300 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.079268    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 21:55:21.152041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.152010    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"
Nov 02 21:55:21.152041 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.152019    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx" event={"ID":"94bd94f6-eeb7-496c-9a76-68ce01b9275a","Type":"ContainerDied","Data":"602055f28ecd7a2a90a021b79bc0436255632bde50b08d32479c57728a62c687"}
Nov 02 21:55:21.152322 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.152074    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="602055f28ecd7a2a90a021b79bc0436255632bde50b08d32479c57728a62c687"
Nov 02 21:55:21.569043 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.569010    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"]
Nov 02 21:55:21.569620 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.569605    4528 cpu_manager.go:414] "containerMap: removing container" podUID="94bd94f6-eeb7-496c-9a76-68ce01b9275a" containerName="revoke-keys"
Nov 02 21:55:21.569670 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.569621    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bd94f6-eeb7-496c-9a76-68ce01b9275a" containerName="revoke-keys"
Nov 02 21:55:21.569736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.569728    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bd94f6-eeb7-496c-9a76-68ce01b9275a" containerName="revoke-keys"
Nov 02 21:55:21.573382 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.573363    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:21.623623 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.623594    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887ce760-4401-437a-a9b5-e1733da74fd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t\" (UID: \"887ce760-4401-437a-a9b5-e1733da74fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:21.624036 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.623699    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cskg\" (UniqueName: \"kubernetes.io/projected/887ce760-4401-437a-a9b5-e1733da74fd7-kube-api-access-4cskg\") pod \"payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t\" (UID: \"887ce760-4401-437a-a9b5-e1733da74fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:21.626527 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.626505    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-58ca7f\""
Nov 02 21:55:21.628080 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.628055    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"]
Nov 02 21:55:21.675449 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.675416    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-58ca7f-dockercfg-kbssg\""
Nov 02 21:55:21.757694 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.757659    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887ce760-4401-437a-a9b5-e1733da74fd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t\" (UID: \"887ce760-4401-437a-a9b5-e1733da74fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:21.758054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.758027    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cskg\" (UniqueName: \"kubernetes.io/projected/887ce760-4401-437a-a9b5-e1733da74fd7-kube-api-access-4cskg\") pod \"payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t\" (UID: \"887ce760-4401-437a-a9b5-e1733da74fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:21.770989 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.770954    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887ce760-4401-437a-a9b5-e1733da74fd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t\" (UID: \"887ce760-4401-437a-a9b5-e1733da74fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:21.782306 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.782266    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"]
Nov 02 21:55:21.806358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.806324    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:21.812136 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.811336    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"]
Nov 02 21:55:21.816380 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.816356    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cskg\" (UniqueName: \"kubernetes.io/projected/887ce760-4401-437a-a9b5-e1733da74fd7-kube-api-access-4cskg\") pod \"payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t\" (UID: \"887ce760-4401-437a-a9b5-e1733da74fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:21.887040 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.886970    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:21.961178 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.961143    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zzfj\" (UniqueName: \"kubernetes.io/projected/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-kube-api-access-9zzfj\") pod \"payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw\" (UID: \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:21.961338 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:21.961223    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw\" (UID: \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:22.062414 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:22.062383    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zzfj\" (UniqueName: \"kubernetes.io/projected/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-kube-api-access-9zzfj\") pod \"payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw\" (UID: \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:22.062598 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:22.062431    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw\" (UID: \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:22.063092 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:22.063073    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw\" (UID: \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:22.119078 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:22.119041    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zzfj\" (UniqueName: \"kubernetes.io/projected/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-kube-api-access-9zzfj\") pod \"payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw\" (UID: \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:22.162600 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:22.162543    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:22.265312 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:22.264984    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"]
Nov 02 21:55:22.535219 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:22.535189    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ad4b49_ac5e_41cf_902f_09c529d3b6dc.slice/crio-49970b45f5324df5ac119a63ff15bee40c4fe88f543e132e390caa8f42a1ce12 WatchSource:0}: Error finding container 49970b45f5324df5ac119a63ff15bee40c4fe88f543e132e390caa8f42a1ce12: Status 404 returned error can't find the container with id 49970b45f5324df5ac119a63ff15bee40c4fe88f543e132e390caa8f42a1ce12
Nov 02 21:55:22.539901 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:22.539867    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"]
Nov 02 21:55:23.165333 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:23.165300    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t" event={"ID":"887ce760-4401-437a-a9b5-e1733da74fd7","Type":"ContainerStarted","Data":"153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329"}
Nov 02 21:55:23.165751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:23.165352    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t" event={"ID":"887ce760-4401-437a-a9b5-e1733da74fd7","Type":"ContainerStarted","Data":"5a172c5440c24a1231cf673ee48c63bb811cea18a38e1d50ff10f2d773f7954f"}
Nov 02 21:55:23.165751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:23.165391    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:23.167095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:23.167070    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw" event={"ID":"50ad4b49-ac5e-41cf-902f-09c529d3b6dc","Type":"ContainerStarted","Data":"a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b"}
Nov 02 21:55:23.167276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:23.167131    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:23.167276 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:23.167154    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw" event={"ID":"50ad4b49-ac5e-41cf-902f-09c529d3b6dc","Type":"ContainerStarted","Data":"49970b45f5324df5ac119a63ff15bee40c4fe88f543e132e390caa8f42a1ce12"}
Nov 02 21:55:23.279029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:23.278985    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t" podStartSLOduration=2.278972861 podStartE2EDuration="2.278972861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:23.277957259 +0000 UTC m=+2810.541493577" watchObservedRunningTime="2026-11-02 21:55:23.278972861 +0000 UTC m=+2810.542509178"
Nov 02 21:55:23.436831 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:23.436706    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw" podStartSLOduration=2.4366874210000002 podStartE2EDuration="2.436687421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:23.426539523 +0000 UTC m=+2810.690075842" watchObservedRunningTime="2026-11-02 21:55:23.436687421 +0000 UTC m=+2810.700223739"
Nov 02 21:55:27.123966 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:27.123935    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 21:55:27.124364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:27.123998    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 21:55:28.993716 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:28.993683    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"]
Nov 02 21:55:28.994120 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:28.993889    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy" containerID="cri-o://5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c" gracePeriod=30
Nov 02 21:55:29.049007 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:29.048982    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"]
Nov 02 21:55:29.052412 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:29.052390    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b21fdc-9dcsw"]
Nov 02 21:55:30.959978 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:30.959932    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5412d7f-dfae-4462-be30-b8e11967fb58" path="/var/lib/kubelet/pods/d5412d7f-dfae-4462-be30-b8e11967fb58/volumes"
Nov 02 21:55:34.181561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.180463    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 21:55:34.183209 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.182444    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 21:55:34.268035 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.268011    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:55:34.269676 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.269644    4528 generic.go:356] "Generic (PLEG): container finished" podID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerID="5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c" exitCode=0
Nov 02 21:55:34.269795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.269687    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" event={"ID":"c8a95916-ee75-44bc-be9e-8cc8baed5bf1","Type":"ContainerDied","Data":"5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c"}
Nov 02 21:55:34.269795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.269751    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm" event={"ID":"c8a95916-ee75-44bc-be9e-8cc8baed5bf1","Type":"ContainerDied","Data":"af3286bf4756fef8a0407a09e60157489ee61009895656170fad1017d370ef12"}
Nov 02 21:55:34.269795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.269777    4528 scope.go:122] "RemoveContainer" containerID="5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c"
Nov 02 21:55:34.282761 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.282739    4528 scope.go:122] "RemoveContainer" containerID="538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c"
Nov 02 21:55:34.292961 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.292937    4528 scope.go:122] "RemoveContainer" containerID="5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c"
Nov 02 21:55:34.293322 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:55:34.293298    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c\": container with ID starting with 5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c not found: ID does not exist" containerID="5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c"
Nov 02 21:55:34.293428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.293329    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c"} err="failed to get container status \"5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c\": rpc error: code = NotFound desc = could not find container \"5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c\": container with ID starting with 5cf5a905fc5821c51df57a9f5ea4e43be77ff2ed7e1d3dc4adcbdecf0b06dc9c not found: ID does not exist"
Nov 02 21:55:34.293428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.293346    4528 scope.go:122] "RemoveContainer" containerID="538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c"
Nov 02 21:55:34.293658 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:55:34.293630    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c\": container with ID starting with 538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c not found: ID does not exist" containerID="538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c"
Nov 02 21:55:34.293714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.293669    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c"} err="failed to get container status \"538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c\": rpc error: code = NotFound desc = could not find container \"538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c\": container with ID starting with 538f10315659466b5f60c4bf8c0529d2a0c9f8dc9ae69c9ce4007aa31fd8906c not found: ID does not exist"
Nov 02 21:55:34.358626 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358593    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-credential-socket\") pod \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") "
Nov 02 21:55:34.358819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358649    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-data\") pod \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") "
Nov 02 21:55:34.358819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358699    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-certs\") pod \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") "
Nov 02 21:55:34.358819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358755    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-kube-api-access-dvdr9\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-kube-api-access-dvdr9\") pod \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") "
Nov 02 21:55:34.358979 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358826    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-token\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-token\") pod \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") "
Nov 02 21:55:34.358979 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358877    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-socket\") pod \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") "
Nov 02 21:55:34.358979 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358892    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-credential-socket" pod "c8a95916-ee75-44bc-be9e-8cc8baed5bf1" (UID: "c8a95916-ee75-44bc-be9e-8cc8baed5bf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:55:34.358979 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358912    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istiod-ca-cert\") pod \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") "
Nov 02 21:55:34.358979 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358943    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-envoy\") pod \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") "
Nov 02 21:55:34.358979 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.358946    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-data" pod "c8a95916-ee75-44bc-be9e-8cc8baed5bf1" (UID: "c8a95916-ee75-44bc-be9e-8cc8baed5bf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:55:34.359252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.359037    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-podinfo\") pod \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\" (UID: \"c8a95916-ee75-44bc-be9e-8cc8baed5bf1\") "
Nov 02 21:55:34.359252 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.359140    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-socket" pod "c8a95916-ee75-44bc-be9e-8cc8baed5bf1" (UID: "c8a95916-ee75-44bc-be9e-8cc8baed5bf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:55:34.359367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.359260    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istiod-ca-cert" pod "c8a95916-ee75-44bc-be9e-8cc8baed5bf1" (UID: "c8a95916-ee75-44bc-be9e-8cc8baed5bf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:55:34.359367 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.359339    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-certs" pod "c8a95916-ee75-44bc-be9e-8cc8baed5bf1" (UID: "c8a95916-ee75-44bc-be9e-8cc8baed5bf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:55:34.359674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.359653    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:34.359674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.359677    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:34.359846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.359692    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:34.359846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.359707    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:34.359846 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.359720    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:34.361163 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.361093    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-token" pod "c8a95916-ee75-44bc-be9e-8cc8baed5bf1" (UID: "c8a95916-ee75-44bc-be9e-8cc8baed5bf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:55:34.364244 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.364218    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-podinfo" pod "c8a95916-ee75-44bc-be9e-8cc8baed5bf1" (UID: "c8a95916-ee75-44bc-be9e-8cc8baed5bf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:55:34.364342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.364237    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-envoy" pod "c8a95916-ee75-44bc-be9e-8cc8baed5bf1" (UID: "c8a95916-ee75-44bc-be9e-8cc8baed5bf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:55:34.364342 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.364311    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-kube-api-access-dvdr9" pod "c8a95916-ee75-44bc-be9e-8cc8baed5bf1" (UID: "c8a95916-ee75-44bc-be9e-8cc8baed5bf1"). InnerVolumeSpecName "kube-api-access-dvdr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:55:34.461287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.461248    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdr9\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-kube-api-access-dvdr9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:34.461287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.461278    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:34.461287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.461293    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:34.461523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:34.461305    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a95916-ee75-44bc-be9e-8cc8baed5bf1-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:55:35.276118 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:35.276085    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"
Nov 02 21:55:35.511625 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:35.511592    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"]
Nov 02 21:55:35.515221 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:35.515187    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3c7706-openshift-default-b589b7fb5-pjnkm"]
Nov 02 21:55:36.914375 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:36.914340    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" path="/var/lib/kubelet/pods/c8a95916-ee75-44bc-be9e-8cc8baed5bf1/volumes"
Nov 02 21:55:37.712311 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.712277    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"]
Nov 02 21:55:37.712871 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.712858    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy"
Nov 02 21:55:37.712911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.712872    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy"
Nov 02 21:55:37.712911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.712887    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy"
Nov 02 21:55:37.712911 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.712894    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy"
Nov 02 21:55:37.713029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.712992    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy"
Nov 02 21:55:37.717996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.717978    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:37.783920 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.783874    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"]
Nov 02 21:55:37.829751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.829002    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnm7\" (UniqueName: \"kubernetes.io/projected/5c199224-f5a0-4200-8ed3-f807671f0416-kube-api-access-6rnm7\") pod \"payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk\" (UID: \"5c199224-f5a0-4200-8ed3-f807671f0416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:37.829751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.829138    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c199224-f5a0-4200-8ed3-f807671f0416-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk\" (UID: \"5c199224-f5a0-4200-8ed3-f807671f0416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:37.829751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.829596    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-06e461\""
Nov 02 21:55:37.829751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.829716    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-06e461-dockercfg-6xz2z\""
Nov 02 21:55:37.876378 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.876334    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"]
Nov 02 21:55:37.877836 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.877800    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a95916-ee75-44bc-be9e-8cc8baed5bf1" containerName="istio-proxy"
Nov 02 21:55:37.883438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.883419    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:37.931235 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.931122    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnm7\" (UniqueName: \"kubernetes.io/projected/5c199224-f5a0-4200-8ed3-f807671f0416-kube-api-access-6rnm7\") pod \"payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk\" (UID: \"5c199224-f5a0-4200-8ed3-f807671f0416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:37.931884 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.931861    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c199224-f5a0-4200-8ed3-f807671f0416-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk\" (UID: \"5c199224-f5a0-4200-8ed3-f807671f0416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:37.933048 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.933025    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c199224-f5a0-4200-8ed3-f807671f0416-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk\" (UID: \"5c199224-f5a0-4200-8ed3-f807671f0416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:37.953095 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.953068    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"]
Nov 02 21:55:37.990685 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:37.990612    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnm7\" (UniqueName: \"kubernetes.io/projected/5c199224-f5a0-4200-8ed3-f807671f0416-kube-api-access-6rnm7\") pod \"payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk\" (UID: \"5c199224-f5a0-4200-8ed3-f807671f0416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:38.030957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.030929    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:38.033883 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.033858    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4bz\" (UniqueName: \"kubernetes.io/projected/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-kube-api-access-dx4bz\") pod \"payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk\" (UID: \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:38.034168 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.034069    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk\" (UID: \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:38.135390 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.135353    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk\" (UID: \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:38.135558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.135495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4bz\" (UniqueName: \"kubernetes.io/projected/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-kube-api-access-dx4bz\") pod \"payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk\" (UID: \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:38.136151 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.136097    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk\" (UID: \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:38.192340 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.192310    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4bz\" (UniqueName: \"kubernetes.io/projected/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-kube-api-access-dx4bz\") pod \"payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk\" (UID: \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:38.213315 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.213292    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:38.399784 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:38.399752    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c199224_f5a0_4200_8ed3_f807671f0416.slice/crio-1fd59d8aca1717fa6b87b8fd5a84fdeda4182b32497c38e42be617bc72af8b3f WatchSource:0}: Error finding container 1fd59d8aca1717fa6b87b8fd5a84fdeda4182b32497c38e42be617bc72af8b3f: Status 404 returned error can't find the container with id 1fd59d8aca1717fa6b87b8fd5a84fdeda4182b32497c38e42be617bc72af8b3f
Nov 02 21:55:38.404343 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.404316    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"]
Nov 02 21:55:38.580319 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:38.580287    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e4a60f_9bee_49b9_b5ac_57598df1aa99.slice/crio-fcbb339915be35870d65e009ef32810b5419501a4f45b87f8ba1fd21b543e043 WatchSource:0}: Error finding container fcbb339915be35870d65e009ef32810b5419501a4f45b87f8ba1fd21b543e043: Status 404 returned error can't find the container with id fcbb339915be35870d65e009ef32810b5419501a4f45b87f8ba1fd21b543e043
Nov 02 21:55:38.583904 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:38.583884    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"]
Nov 02 21:55:39.104727 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:39.104691    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:55:39.305461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:39.305429    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk" event={"ID":"e7e4a60f-9bee-49b9-b5ac-57598df1aa99","Type":"ContainerStarted","Data":"1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6"}
Nov 02 21:55:39.305659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:39.305488    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk" event={"ID":"e7e4a60f-9bee-49b9-b5ac-57598df1aa99","Type":"ContainerStarted","Data":"fcbb339915be35870d65e009ef32810b5419501a4f45b87f8ba1fd21b543e043"}
Nov 02 21:55:39.305659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:39.305518    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:39.307192 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:39.307168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk" event={"ID":"5c199224-f5a0-4200-8ed3-f807671f0416","Type":"ContainerStarted","Data":"426c2d461e5c17253066751d72020e3a62986c8287dd8ec68d60824618d876a9"}
Nov 02 21:55:39.307314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:39.307206    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk" event={"ID":"5c199224-f5a0-4200-8ed3-f807671f0416","Type":"ContainerStarted","Data":"1fd59d8aca1717fa6b87b8fd5a84fdeda4182b32497c38e42be617bc72af8b3f"}
Nov 02 21:55:39.307314 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:39.307252    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:39.419567 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:39.419475    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk" podStartSLOduration=2.419461315 podStartE2EDuration="2.419461315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:39.418017514 +0000 UTC m=+2826.681553832" watchObservedRunningTime="2026-11-02 21:55:39.419461315 +0000 UTC m=+2826.682997646"
Nov 02 21:55:39.578856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:39.578795    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk" podStartSLOduration=2.578781875 podStartE2EDuration="2.578781875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:39.575318227 +0000 UTC m=+2826.838854545" watchObservedRunningTime="2026-11-02 21:55:39.578781875 +0000 UTC m=+2826.842318192"
Nov 02 21:55:41.051189 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:41.051160    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"]
Nov 02 21:55:41.099976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:41.099940    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea45e1b-66t4b"]
Nov 02 21:55:42.301908 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.301877    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"]
Nov 02 21:55:42.307362 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.307341    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:42.311909 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.311884    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v\" (UID: \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:42.312020 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.312003    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmd4t\" (UniqueName: \"kubernetes.io/projected/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-kube-api-access-mmd4t\") pod \"payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v\" (UID: \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:42.406180 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.404238    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"]
Nov 02 21:55:42.417953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.413910    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v\" (UID: \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:42.417953 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.414045    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmd4t\" (UniqueName: \"kubernetes.io/projected/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-kube-api-access-mmd4t\") pod \"payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v\" (UID: \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:42.484894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.475444    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-4ca09f-dockercfg-9kb2r\""
Nov 02 21:55:42.484894 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.475509    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-4ca09f\""
Nov 02 21:55:42.487447 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.487413    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v\" (UID: \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:42.493354 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.493322    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmd4t\" (UniqueName: \"kubernetes.io/projected/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-kube-api-access-mmd4t\") pod \"payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v\" (UID: \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:42.495941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.495914    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"]
Nov 02 21:55:42.506624 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.505715    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:42.531714 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.531684    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"]
Nov 02 21:55:42.618324 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.618243    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m98r\" (UniqueName: \"kubernetes.io/projected/2c2f8896-050b-4a6b-a685-2975ac72b63f-kube-api-access-7m98r\") pod \"payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7\" (UID: \"2c2f8896-050b-4a6b-a685-2975ac72b63f\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:42.618480 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.618419    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2f8896-050b-4a6b-a685-2975ac72b63f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7\" (UID: \"2c2f8896-050b-4a6b-a685-2975ac72b63f\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:42.623064 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.623042    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:42.719382 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.719332    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m98r\" (UniqueName: \"kubernetes.io/projected/2c2f8896-050b-4a6b-a685-2975ac72b63f-kube-api-access-7m98r\") pod \"payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7\" (UID: \"2c2f8896-050b-4a6b-a685-2975ac72b63f\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:42.719558 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.719536    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2f8896-050b-4a6b-a685-2975ac72b63f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7\" (UID: \"2c2f8896-050b-4a6b-a685-2975ac72b63f\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:42.720336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.720311    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2f8896-050b-4a6b-a685-2975ac72b63f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7\" (UID: \"2c2f8896-050b-4a6b-a685-2975ac72b63f\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:42.776400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.776377    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m98r\" (UniqueName: \"kubernetes.io/projected/2c2f8896-050b-4a6b-a685-2975ac72b63f-kube-api-access-7m98r\") pod \"payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7\" (UID: \"2c2f8896-050b-4a6b-a685-2975ac72b63f\") " pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:42.827877 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.827854    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:42.916441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.916405    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123c5c7b-2fc5-4875-b20f-f532461bd641" path="/var/lib/kubelet/pods/123c5c7b-2fc5-4875-b20f-f532461bd641/volumes"
Nov 02 21:55:42.993610 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:42.993576    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ea82f2_85a7_4e4a_bd18_1ef77dacb533.slice/crio-25114381f3a4848576abe1631fb026a5fc50d9dc810cb1c0cf812e0e8892982b WatchSource:0}: Error finding container 25114381f3a4848576abe1631fb026a5fc50d9dc810cb1c0cf812e0e8892982b: Status 404 returned error can't find the container with id 25114381f3a4848576abe1631fb026a5fc50d9dc810cb1c0cf812e0e8892982b
Nov 02 21:55:42.997385 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:42.997364    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"]
Nov 02 21:55:43.195277 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:55:43.195251    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2f8896_050b_4a6b_a685_2975ac72b63f.slice/crio-12d269223773b61ef62a35eea9722e997c6f831dc28bf883ec70685f990e6ea8 WatchSource:0}: Error finding container 12d269223773b61ef62a35eea9722e997c6f831dc28bf883ec70685f990e6ea8: Status 404 returned error can't find the container with id 12d269223773b61ef62a35eea9722e997c6f831dc28bf883ec70685f990e6ea8
Nov 02 21:55:43.199423 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:43.199401    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"]
Nov 02 21:55:43.334428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:43.334398    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7" event={"ID":"2c2f8896-050b-4a6b-a685-2975ac72b63f","Type":"ContainerStarted","Data":"2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499"}
Nov 02 21:55:43.334831 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:43.334459    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:43.334831 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:43.334481    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7" event={"ID":"2c2f8896-050b-4a6b-a685-2975ac72b63f","Type":"ContainerStarted","Data":"12d269223773b61ef62a35eea9722e997c6f831dc28bf883ec70685f990e6ea8"}
Nov 02 21:55:43.336072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:43.336050    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v" event={"ID":"08ea82f2-85a7-4e4a-bd18-1ef77dacb533","Type":"ContainerStarted","Data":"22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b"}
Nov 02 21:55:43.336198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:43.336092    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v" event={"ID":"08ea82f2-85a7-4e4a-bd18-1ef77dacb533","Type":"ContainerStarted","Data":"25114381f3a4848576abe1631fb026a5fc50d9dc810cb1c0cf812e0e8892982b"}
Nov 02 21:55:43.336198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:43.336149    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:43.451268 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:43.451186    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7" podStartSLOduration=1.451173184 podStartE2EDuration="1.451173184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:43.448841185 +0000 UTC m=+2830.712377495" watchObservedRunningTime="2026-11-02 21:55:43.451173184 +0000 UTC m=+2830.714709539"
Nov 02 21:55:43.590646 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:43.588865    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v" podStartSLOduration=1.588843095 podStartE2EDuration="1.588843095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:43.569554997 +0000 UTC m=+2830.833091324" watchObservedRunningTime="2026-11-02 21:55:43.588843095 +0000 UTC m=+2830.852379413"
Nov 02 21:55:44.113499 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:55:44.113466    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:55:45.047991 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:45.047958    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"]
Nov 02 21:55:45.097972 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:45.097943    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ba04d-phv7l"]
Nov 02 21:55:46.914294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:46.914261    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc6c624-333d-4f0d-8938-93c3116e723d" path="/var/lib/kubelet/pods/ecc6c624-333d-4f0d-8938-93c3116e723d/volumes"
Nov 02 21:55:50.317236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:50.317204    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 21:55:50.317662 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:50.317275    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 21:55:54.344790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:54.344756    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 21:55:54.345313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:54.345153    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 21:55:56.056003 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:56.055974    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"]
Nov 02 21:55:56.105559 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:56.105529    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b962dec-f8lxt"]
Nov 02 21:55:56.914604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:56.914572    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd349a2-13f8-40a2-8d0f-996ba522b27f" path="/var/lib/kubelet/pods/acd349a2-13f8-40a2-8d0f-996ba522b27f/volumes"
Nov 02 21:55:58.050566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:58.050530    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"]
Nov 02 21:55:58.101220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:58.101191    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536b0d42-76plt"]
Nov 02 21:55:58.915236 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:55:58.915202    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87b2f26-35bb-41d9-8f9b-6b4d71a22624" path="/var/lib/kubelet/pods/e87b2f26-35bb-41d9-8f9b-6b4d71a22624/volumes"
Nov 02 21:56:13.977250 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:13.977208    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:56:13.977823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:13.977260    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:56:14.112168 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:56:14.112134    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:56:14.603016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:14.602981    4528 generic.go:356] "Generic (PLEG): container finished" podID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerID="123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a" exitCode=0
Nov 02 21:56:14.603281 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:14.603057    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" event={"ID":"1adfc106-9a49-44d2-b5c7-cca45733f6bd","Type":"ContainerDied","Data":"123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a"}
Nov 02 21:56:14.603524 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:14.603506    4528 scope.go:122] "RemoveContainer" containerID="123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a"
Nov 02 21:56:14.605093 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:14.605067    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:56:14.605182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:14.605164    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:56:14.605220 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:14.605199    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:56:15.633657 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:15.633622    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" event={"ID":"1adfc106-9a49-44d2-b5c7-cca45733f6bd","Type":"ContainerStarted","Data":"9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d"}
Nov 02 21:56:16.563170 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:16.563134    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:56:16.563353 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:16.563260    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 21:56:16.563353 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:16.563307    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 21:56:17.048355 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:17.048327    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"]
Nov 02 21:56:17.098090 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:17.098065    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e882f3d1-bxwzr"]
Nov 02 21:56:17.563336 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:17.563302    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 21:56:17.563510 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:17.563352    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 21:56:18.562910 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:18.562876    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 21:56:18.563332 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:18.562930    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 21:56:18.930566 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:18.930480    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0" path="/var/lib/kubelet/pods/f11f34d2-0d88-4cb5-9df7-31f29d0dd5b0/volumes"
Nov 02 21:56:19.562665 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:19.562631    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 21:56:19.562856 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:19.562695    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 21:56:20.574331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:20.574300    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:56:20.574751 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:20.574731    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:56:20.577628 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:20.577607    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 21:56:22.048016 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:22.047985    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"]
Nov 02 21:56:22.098274 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:22.098246    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f4cd4d-xxgsc"]
Nov 02 21:56:22.914771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:22.914738    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008260e5-f324-4292-8a55-1872010e6b36" path="/var/lib/kubelet/pods/008260e5-f324-4292-8a55-1872010e6b36/volumes"
Nov 02 21:56:33.955631 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:33.955598    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:56:34.994174 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:34.994143    4528 scope.go:122] "RemoveContainer" containerID="f8ba52fd8121ecffa9b7f3acb29a77a4c495dd586745f6ef021d40c0260b9860"
Nov 02 21:56:35.008628 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:35.008601    4528 scope.go:122] "RemoveContainer" containerID="4117ebfca3e21bcb05a7d8d929d45504c2c082ad2e199c38a4847e7e4acc1161"
Nov 02 21:56:35.018792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:35.018766    4528 scope.go:122] "RemoveContainer" containerID="dc561ecb6132b11a6d7bb87c168b6e4eeae66f110cfe210cd775cbbe545a3cd0"
Nov 02 21:56:35.027355 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:35.027333    4528 scope.go:122] "RemoveContainer" containerID="9334722ba3c961c1c344d86d6acf6ff7fe2c08e9a0ca10418b7524c6712f9022"
Nov 02 21:56:41.049670 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:41.049640    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"]
Nov 02 21:56:41.099868 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:41.099835    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf2a166-28r89"]
Nov 02 21:56:42.915796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:42.915758    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01cdaed5-23e4-43e2-9f09-11a4230b270c" path="/var/lib/kubelet/pods/01cdaed5-23e4-43e2-9f09-11a4230b270c/volumes"
Nov 02 21:56:44.115638 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:56:44.115601    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:56:44.888088 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:44.888047    4528 generic.go:356] "Generic (PLEG): container finished" podID="387be747-622a-4118-89c1-9565f9c5e947" containerID="37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1" exitCode=0
Nov 02 21:56:44.888284 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:44.888129    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" event={"ID":"387be747-622a-4118-89c1-9565f9c5e947","Type":"ContainerDied","Data":"37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1"}
Nov 02 21:56:44.888621 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:44.888603    4528 scope.go:122] "RemoveContainer" containerID="37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1"
Nov 02 21:56:44.890287 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:44.890261    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:56:44.890370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:44.890330    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:56:44.890370 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:44.890363    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:56:45.900262 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:45.900223    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" event={"ID":"387be747-622a-4118-89c1-9565f9c5e947","Type":"ContainerStarted","Data":"30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5"}
Nov 02 21:56:46.457264 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:46.457226    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:56:46.457454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:46.457384    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 21:56:46.457454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:46.457427    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 21:56:47.457472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:47.457438    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 21:56:47.457958 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:47.457510    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 21:56:48.047211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:48.047178    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"]
Nov 02 21:56:48.061479 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:48.061453    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10bcb25a-hz969"]
Nov 02 21:56:48.456818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:48.456731    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 21:56:48.456818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:48.456804    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 21:56:48.932710 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:48.932675    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49825c7e-57c9-43c1-a274-f0f948561807" path="/var/lib/kubelet/pods/49825c7e-57c9-43c1-a274-f0f948561807/volumes"
Nov 02 21:56:49.457750 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:49.457717    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 21:56:49.457937 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:49.457780    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 21:56:50.457461 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:50.457429    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 21:56:50.457857 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:50.457492    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 21:56:51.461024 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:51.460990    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:56:51.461445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:51.461424    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:56:51.462101 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:51.462082    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 21:56:56.047792 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:56.047753    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"]
Nov 02 21:56:56.097638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:56.097609    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0022265-vq7md"]
Nov 02 21:56:56.914441 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:56:56.914407    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7" path="/var/lib/kubelet/pods/a1a9c036-44f7-4fe0-9d54-2a4f3e32e1a7/volumes"
Nov 02 21:57:01.180528 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:01.180489    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:57:03.094904 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:03.094871    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"]
Nov 02 21:57:03.098135 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:03.098097    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6ccca7-7mpbg"]
Nov 02 21:57:04.914339 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:04.914308    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9ed414-d247-480a-82d4-56c3621a6b94" path="/var/lib/kubelet/pods/3c9ed414-d247-480a-82d4-56c3621a6b94/volumes"
Nov 02 21:57:14.115747 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:57:14.115709    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:57:17.526790 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.526757    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"]
Nov 02 21:57:17.531261 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.531244    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.552708 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.552683    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-model-cache\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.552815 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.552781    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-home\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.552862 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.552823    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tmp-dir\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.552942 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.552923    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xxd\" (UniqueName: \"kubernetes.io/projected/197d7371-cd9c-4fc4-959e-d319d38cf7dc-kube-api-access-z8xxd\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.552996 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.552981    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tls-certs\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.553072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.553051    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-dshm\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.585723 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.585688    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"]
Nov 02 21:57:17.630334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.630304    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-97995b\"/\"openshift-service-ca.crt\""
Nov 02 21:57:17.630334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.630316    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-97995b\"/\"default-dockercfg-h5psl\""
Nov 02 21:57:17.630494 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.630345    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-97995b\"/\"auth-test-model-97995b-kserve-self-signed-certs\""
Nov 02 21:57:17.634013 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.633998    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-97995b\"/\"kube-root-ca.crt\""
Nov 02 21:57:17.653864 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.653843    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tls-certs\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.653943 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.653884    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-dshm\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.654006 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.653991    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-model-cache\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.654061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.654048    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-home\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.654131 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.654093    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tmp-dir\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.654193 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.654162    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xxd\" (UniqueName: \"kubernetes.io/projected/197d7371-cd9c-4fc4-959e-d319d38cf7dc-kube-api-access-z8xxd\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.654472 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.654452    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-model-cache\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.654563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.654496    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-home\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.654563 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.654527    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tmp-dir\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.656211 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.656190    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-dshm\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.656422 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.656403    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tls-certs\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.717464 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.717432    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xxd\" (UniqueName: \"kubernetes.io/projected/197d7371-cd9c-4fc4-959e-d319d38cf7dc-kube-api-access-z8xxd\") pod \"auth-test-model-97995b-kserve-74445cfd57-hndzz\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") " pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:17.843155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:17.843065    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:18.217823 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:18.217743    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"]
Nov 02 21:57:18.230012 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:57:18.229985    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod197d7371_cd9c_4fc4_959e_d319d38cf7dc.slice/crio-66deb0a4f6ccc386ce772778bdc00720b8c2c04fae1662f56e3ddd36d3bc631d WatchSource:0}: Error finding container 66deb0a4f6ccc386ce772778bdc00720b8c2c04fae1662f56e3ddd36d3bc631d: Status 404 returned error can't find the container with id 66deb0a4f6ccc386ce772778bdc00720b8c2c04fae1662f56e3ddd36d3bc631d
Nov 02 21:57:19.056256 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:19.056224    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"]
Nov 02 21:57:19.105960 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:19.105927    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"]
Nov 02 21:57:19.108925 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:19.108899    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ad956c-29727"]
Nov 02 21:57:19.111465 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:19.111445    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ed5ce0-d2528"]
Nov 02 21:57:19.196187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:19.196151    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz" event={"ID":"197d7371-cd9c-4fc4-959e-d319d38cf7dc","Type":"ContainerStarted","Data":"64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c"}
Nov 02 21:57:19.196356 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:19.196198    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz" event={"ID":"197d7371-cd9c-4fc4-959e-d319d38cf7dc","Type":"ContainerStarted","Data":"66deb0a4f6ccc386ce772778bdc00720b8c2c04fae1662f56e3ddd36d3bc631d"}
Nov 02 21:57:19.313313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:19.313218    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz" podStartSLOduration=2.313203333 podStartE2EDuration="2.313203333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:19.310038034 +0000 UTC m=+2926.573574352" watchObservedRunningTime="2026-11-02 21:57:19.313203333 +0000 UTC m=+2926.576739650"
Nov 02 21:57:19.843818 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:19.843786    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:20.913921 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:20.913890    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5b2434-443f-4a04-8275-5e1a78f967c7" path="/var/lib/kubelet/pods/9b5b2434-443f-4a04-8275-5e1a78f967c7/volumes"
Nov 02 21:57:20.914334 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:20.914312    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a20006e5-e52b-4756-a828-20cfbd1ad818" path="/var/lib/kubelet/pods/a20006e5-e52b-4756-a828-20cfbd1ad818/volumes"
Nov 02 21:57:25.047280 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:25.047248    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"]
Nov 02 21:57:25.108171 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:25.099180    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225d1433-vq8rz"]
Nov 02 21:57:26.914771 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:26.914741    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a40bb762-0d41-490d-baac-b963dfd30fd5" path="/var/lib/kubelet/pods/a40bb762-0d41-490d-baac-b963dfd30fd5/volumes"
Nov 02 21:57:27.843517 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:27.843473    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:27.855604 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:27.855582    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:28.271659 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:28.271634    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 21:57:30.095796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:30.095764    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"]
Nov 02 21:57:30.098597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:30.098576    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8802b5-mjc9c"]
Nov 02 21:57:30.916545 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:30.916508    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87471c0-823c-454f-8bbd-e976b1f7d3fe" path="/var/lib/kubelet/pods/d87471c0-823c-454f-8bbd-e976b1f7d3fe/volumes"
Nov 02 21:57:34.046483 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:34.046448    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"]
Nov 02 21:57:34.096754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:34.096724    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03be370e-v252m"]
Nov 02 21:57:34.682350 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:34.682316    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:57:34.915029 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:34.914997    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3423c69-3870-44e7-aee8-8d30cfe87bf6" path="/var/lib/kubelet/pods/f3423c69-3870-44e7-aee8-8d30cfe87bf6/volumes"
Nov 02 21:57:35.073504 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:35.073476    4528 scope.go:122] "RemoveContainer" containerID="6dd0f2d5e156c99b2278b0ad071133aa67acb96c83d71b975c57608ebd40eadd"
Nov 02 21:57:35.084086 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:35.084066    4528 scope.go:122] "RemoveContainer" containerID="ec4b7825f8dd8df4683a630ef74cfab02854d81f35b6be4a0ca52268252a1612"
Nov 02 21:57:35.093374 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:35.093356    4528 scope.go:122] "RemoveContainer" containerID="480f4337282cc20603b65b407e61a88bfc96e41825a45fa1d170149c63f6127b"
Nov 02 21:57:35.102027 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:35.102001    4528 scope.go:122] "RemoveContainer" containerID="497f4a45516e9490b272fc2645fec2a49d03590deade5adb138056fe5e91d4ae"
Nov 02 21:57:35.110853 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:35.110834    4528 scope.go:122] "RemoveContainer" containerID="60b96bdde435107b85a574d6f4af8396ec66d37330abbc3cbbac292cb964ae25"
Nov 02 21:57:35.121364 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:35.121347    4528 scope.go:122] "RemoveContainer" containerID="42035cfb090eb971e62e36d0847a23a93905c9d28a4c3c17e7122be2fb72619f"
Nov 02 21:57:36.049397 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:36.049362    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"]
Nov 02 21:57:36.108331 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:36.108290    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41bd83-ktr4r"]
Nov 02 21:57:36.915640 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:36.915611    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10c5b99-4438-4952-99aa-6a49a69f5de4" path="/var/lib/kubelet/pods/c10c5b99-4438-4952-99aa-6a49a69f5de4/volumes"
Nov 02 21:57:44.114057 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:57:44.114022    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:57:51.051814 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:51.051781    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"]
Nov 02 21:57:51.098305 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:51.098276    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a873435-8cmpq"]
Nov 02 21:57:52.914187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:52.914154    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c91b35c-d00f-4dbb-a601-b0f865238201" path="/var/lib/kubelet/pods/9c91b35c-d00f-4dbb-a601-b0f865238201/volumes"
Nov 02 21:57:53.094774 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:53.094741    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"]
Nov 02 21:57:53.097642 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:53.097616    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9b6a9c-8ftkq"]
Nov 02 21:57:54.913486 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:57:54.913454    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a65a47e-f350-4381-90d6-7aab121bd723" path="/var/lib/kubelet/pods/5a65a47e-f350-4381-90d6-7aab121bd723/volumes"
Nov 02 21:58:03.041967 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.041937    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"]
Nov 02 21:58:03.046011 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.045990    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.113132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.104171    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c7e71f-openshift-default-dockercfg-nzlrp\""
Nov 02 21:58:03.202191 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.161896    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfvf\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-kube-api-access-zpfvf\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.202633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.202571    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-credential-socket\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.202891 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.202869    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-socket\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.203217 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.203184    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-certs\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.204365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.204338    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-data\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.204633 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.204613    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-podinfo\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.204803 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.204788    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474a4c1-c212-4544-8e0c-f9e411b5b219-istiod-ca-cert\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.204974 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.204961    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-envoy\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.205132 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.205103    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-token\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.232736 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.232704    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"]
Nov 02 21:58:03.307142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307021    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-credential-socket\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307142 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307067    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-socket\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307150    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-certs\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307191    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-data\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307222    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-podinfo\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307244    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474a4c1-c212-4544-8e0c-f9e411b5b219-istiod-ca-cert\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307295    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-envoy\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307322    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-token\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfvf\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-kube-api-access-zpfvf\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307551 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307491    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-credential-socket\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307659    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-socket\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.307977 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.307712    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-certs\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.308079 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.308026    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-data\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.308432 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.308414    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474a4c1-c212-4544-8e0c-f9e411b5b219-istiod-ca-cert\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.309870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.309831    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-podinfo\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.309870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.309854    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-envoy\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.380388 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.375732    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfvf\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-kube-api-access-zpfvf\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.399166 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.399131    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-token\") pod \"e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") " pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:03.660304 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:03.660228    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:04.082655 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:58:04.082612    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd474a4c1_c212_4544_8e0c_f9e411b5b219.slice/crio-1eff552549c0fb03d62291dcac00d29cf834a57d5f29a9b05f16c2a5d15d12b2 WatchSource:0}: Error finding container 1eff552549c0fb03d62291dcac00d29cf834a57d5f29a9b05f16c2a5d15d12b2: Status 404 returned error can't find the container with id 1eff552549c0fb03d62291dcac00d29cf834a57d5f29a9b05f16c2a5d15d12b2
Nov 02 21:58:04.085637 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:04.085601    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:58:04.085755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:04.085683    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:58:04.085755 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:04.085725    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:58:04.085941 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:04.085915    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"]
Nov 02 21:58:04.503674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:04.503599    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" event={"ID":"d474a4c1-c212-4544-8e0c-f9e411b5b219","Type":"ContainerStarted","Data":"723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117"}
Nov 02 21:58:04.503674 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:04.503660    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" event={"ID":"d474a4c1-c212-4544-8e0c-f9e411b5b219","Type":"ContainerStarted","Data":"1eff552549c0fb03d62291dcac00d29cf834a57d5f29a9b05f16c2a5d15d12b2"}
Nov 02 21:58:04.625795 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:04.625744    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" podStartSLOduration=2.625731681 podStartE2EDuration="2.625731681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:04.623673848 +0000 UTC m=+2971.887210164" watchObservedRunningTime="2026-11-02 21:58:04.625731681 +0000 UTC m=+2971.889267997"
Nov 02 21:58:04.660534 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:04.660498    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:05.666386 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:05.666354    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:06.516179 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:06.516146    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:06.517022 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:06.517002    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:58:11.015459 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.015422    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"]
Nov 02 21:58:11.019794 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.019778    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:11.071680 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.071653    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"]
Nov 02 21:58:11.094630 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.094602    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c7e71f\""
Nov 02 21:58:11.094793 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.094642    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c7e71f-dockercfg-hxr9g\""
Nov 02 21:58:11.115677 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.115653    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664sl\" (UniqueName: \"kubernetes.io/projected/3a493258-30e3-4d3b-beb0-b91417fd4152-kube-api-access-664sl\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4\" (UID: \"3a493258-30e3-4d3b-beb0-b91417fd4152\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:11.115796 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.115685    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a493258-30e3-4d3b-beb0-b91417fd4152-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4\" (UID: \"3a493258-30e3-4d3b-beb0-b91417fd4152\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:11.223293 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.218166    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"]
Nov 02 21:58:11.231978 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.219227    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664sl\" (UniqueName: \"kubernetes.io/projected/3a493258-30e3-4d3b-beb0-b91417fd4152-kube-api-access-664sl\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4\" (UID: \"3a493258-30e3-4d3b-beb0-b91417fd4152\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:11.232279 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.232256    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a493258-30e3-4d3b-beb0-b91417fd4152-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4\" (UID: \"3a493258-30e3-4d3b-beb0-b91417fd4152\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:11.253133 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.233518    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a493258-30e3-4d3b-beb0-b91417fd4152-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4\" (UID: \"3a493258-30e3-4d3b-beb0-b91417fd4152\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:11.286176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.282564    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:11.286955 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.286917    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"]
Nov 02 21:58:11.294561 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.294481    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-664sl\" (UniqueName: \"kubernetes.io/projected/3a493258-30e3-4d3b-beb0-b91417fd4152-kube-api-access-664sl\") pod \"payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4\" (UID: \"3a493258-30e3-4d3b-beb0-b91417fd4152\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:11.334957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.333825    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad99e9a-8455-4df2-8489-07e0c393d68b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c\" (UID: \"0ad99e9a-8455-4df2-8489-07e0c393d68b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:11.334957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.333913    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdpvk\" (UniqueName: \"kubernetes.io/projected/0ad99e9a-8455-4df2-8489-07e0c393d68b-kube-api-access-pdpvk\") pod \"payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c\" (UID: \"0ad99e9a-8455-4df2-8489-07e0c393d68b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:11.334957 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.334513    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:11.436072 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.436038    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad99e9a-8455-4df2-8489-07e0c393d68b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c\" (UID: \"0ad99e9a-8455-4df2-8489-07e0c393d68b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:11.436444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.436416    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpvk\" (UniqueName: \"kubernetes.io/projected/0ad99e9a-8455-4df2-8489-07e0c393d68b-kube-api-access-pdpvk\") pod \"payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c\" (UID: \"0ad99e9a-8455-4df2-8489-07e0c393d68b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:11.436954 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.436931    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad99e9a-8455-4df2-8489-07e0c393d68b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c\" (UID: \"0ad99e9a-8455-4df2-8489-07e0c393d68b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:11.493216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.493190    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdpvk\" (UniqueName: \"kubernetes.io/projected/0ad99e9a-8455-4df2-8489-07e0c393d68b-kube-api-access-pdpvk\") pod \"payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c\" (UID: \"0ad99e9a-8455-4df2-8489-07e0c393d68b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:11.631301 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.631226    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:11.708780 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:58:11.708746    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a493258_30e3_4d3b_beb0_b91417fd4152.slice/crio-27cfab9d25f43e0d58e00710e0e53486b4f37c807793ee96eb2503a9d3450338 WatchSource:0}: Error finding container 27cfab9d25f43e0d58e00710e0e53486b4f37c807793ee96eb2503a9d3450338: Status 404 returned error can't find the container with id 27cfab9d25f43e0d58e00710e0e53486b4f37c807793ee96eb2503a9d3450338
Nov 02 21:58:11.711700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:11.711676    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"]
Nov 02 21:58:12.001328 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:58:12.001295    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad99e9a_8455_4df2_8489_07e0c393d68b.slice/crio-aeb2db92e6048aeb96df66ee3e937b43ce4c79bcdec2401f62dcdd3800c2d1c7 WatchSource:0}: Error finding container aeb2db92e6048aeb96df66ee3e937b43ce4c79bcdec2401f62dcdd3800c2d1c7: Status 404 returned error can't find the container with id aeb2db92e6048aeb96df66ee3e937b43ce4c79bcdec2401f62dcdd3800c2d1c7
Nov 02 21:58:12.004638 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:12.004618    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"]
Nov 02 21:58:12.553879 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:12.553838    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c" event={"ID":"0ad99e9a-8455-4df2-8489-07e0c393d68b","Type":"ContainerStarted","Data":"22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56"}
Nov 02 21:58:12.554294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:12.553896    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c" event={"ID":"0ad99e9a-8455-4df2-8489-07e0c393d68b","Type":"ContainerStarted","Data":"aeb2db92e6048aeb96df66ee3e937b43ce4c79bcdec2401f62dcdd3800c2d1c7"}
Nov 02 21:58:12.554294 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:12.554030    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:12.555481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:12.555456    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4" event={"ID":"3a493258-30e3-4d3b-beb0-b91417fd4152","Type":"ContainerStarted","Data":"10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53"}
Nov 02 21:58:12.555579 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:12.555493    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4" event={"ID":"3a493258-30e3-4d3b-beb0-b91417fd4152","Type":"ContainerStarted","Data":"27cfab9d25f43e0d58e00710e0e53486b4f37c807793ee96eb2503a9d3450338"}
Nov 02 21:58:12.555579 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:12.555542    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:12.670182 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:12.670133    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c" podStartSLOduration=1.6701042099999999 podStartE2EDuration="1.67010421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:12.668466248 +0000 UTC m=+2979.932002564" watchObservedRunningTime="2026-11-02 21:58:12.67010421 +0000 UTC m=+2979.933640527"
Nov 02 21:58:12.811187 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:12.804519    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4" podStartSLOduration=2.804499446 podStartE2EDuration="2.804499446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:12.790185161 +0000 UTC m=+2980.053721479" watchObservedRunningTime="2026-11-02 21:58:12.804499446 +0000 UTC m=+2980.068035763"
Nov 02 21:58:14.113185 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:58:14.113145    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:58:23.565415 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:23.565379    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 21:58:23.565896 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:23.565520    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 21:58:30.639515 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.639479    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"]
Nov 02 21:58:30.644216 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.644193    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.695406 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.695379    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"]
Nov 02 21:58:30.710744 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.710712    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-home\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.710919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.710871    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tmp-dir\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.710919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.710899    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-dshm\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.711054 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.710923    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-model-cache\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.711158 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.711089    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tls-certs\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.711235 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.711209    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxqkw\" (UniqueName: \"kubernetes.io/projected/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-kube-api-access-bxqkw\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.741919 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.741889    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7cde84\"/\"kube-root-ca.crt\""
Nov 02 21:58:30.742049 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.741930    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7cde84\"/\"default-dockercfg-44ldk\""
Nov 02 21:58:30.747198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.747177    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7cde84\"/\"auth-test-model-7cde84-kserve-self-signed-certs\""
Nov 02 21:58:30.747198 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.747195    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7cde84\"/\"openshift-service-ca.crt\""
Nov 02 21:58:30.816139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.815983    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-model-cache\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.816139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.816072    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tls-certs\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.816395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.816186    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxqkw\" (UniqueName: \"kubernetes.io/projected/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-kube-api-access-bxqkw\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.816395 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.816320    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-home\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.816502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.816442    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tmp-dir\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.816502 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.816477    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-dshm\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.818165 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.818088    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-model-cache\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.819887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.819556    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-home\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.819887 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.819800    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tmp-dir\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.831061 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.831000    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tls-certs\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.839127 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.834715    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-dshm\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.873978 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.873944    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxqkw\" (UniqueName: \"kubernetes.io/projected/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-kube-api-access-bxqkw\") pod \"auth-test-model-7cde84-kserve-65449b46d5-dzb47\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:30.957976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:30.957900    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:31.326541 ip-10-0-133-86 kubenswrapper[4528]: W1102 21:58:31.326507    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0625b31_c3c2_4d5a_9192_4ab8c3d67a32.slice/crio-70f56a26086ea789bbe146a935a59db2f9ac54fc74eccaa8df508c54347b21aa WatchSource:0}: Error finding container 70f56a26086ea789bbe146a935a59db2f9ac54fc74eccaa8df508c54347b21aa: Status 404 returned error can't find the container with id 70f56a26086ea789bbe146a935a59db2f9ac54fc74eccaa8df508c54347b21aa
Nov 02 21:58:31.329325 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:31.329303    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"]
Nov 02 21:58:31.680473 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:31.680380    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47" event={"ID":"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32","Type":"ContainerStarted","Data":"648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17"}
Nov 02 21:58:31.680473 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:31.680427    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47" event={"ID":"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32","Type":"ContainerStarted","Data":"70f56a26086ea789bbe146a935a59db2f9ac54fc74eccaa8df508c54347b21aa"}
Nov 02 21:58:31.794578 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:31.794519    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47" podStartSLOduration=1.794504656 podStartE2EDuration="1.794504656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:31.793713388 +0000 UTC m=+2999.057249706" watchObservedRunningTime="2026-11-02 21:58:31.794504656 +0000 UTC m=+2999.058041015"
Nov 02 21:58:31.960316 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:31.960217    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:35.179484 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.179448    4528 scope.go:122] "RemoveContainer" containerID="e1f55efac5b939d28a5ef3558b79f8c67d0dbfa69cd52f35a56185fb138f646a"
Nov 02 21:58:35.189361 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.189338    4528 scope.go:122] "RemoveContainer" containerID="1911cb92a445fdebf696c3c27c0d6c088a46eea5cef9306f4ade5604cf2366b0"
Nov 02 21:58:35.198139 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.198103    4528 scope.go:122] "RemoveContainer" containerID="96bc46c764375b64e39953e77bd2577ba339d03a36fbeb99aa409174f4dc1dee"
Nov 02 21:58:35.207444 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.207424    4528 scope.go:122] "RemoveContainer" containerID="9f981e378657ea722c81cd54cab4fa3cc2c9f04fa03bbe8d37b8653ffdffae15"
Nov 02 21:58:35.215572 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.215552    4528 scope.go:122] "RemoveContainer" containerID="aac6da1547cb026e2408bfbf2b1c9e374827b2a07ac7124d15bf0042c3ceccae"
Nov 02 21:58:35.223754 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.223736    4528 scope.go:122] "RemoveContainer" containerID="32d85dd25de2cc29420ae588ea74ea9f555ade29982e4ceb2cfaf4a9ee9774c6"
Nov 02 21:58:35.232098 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.232084    4528 scope.go:122] "RemoveContainer" containerID="04c25dc68b66f9e10c47737d021668afd1d55d12f20195a646b55e48367f9e5f"
Nov 02 21:58:35.240155 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.240135    4528 scope.go:122] "RemoveContainer" containerID="2eef7cf059f9be9e2f1f2be6961e657c82c7c22220021db8f7b4150ef288552d"
Nov 02 21:58:35.251700 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.251668    4528 scope.go:122] "RemoveContainer" containerID="5258b5995d7f91b7e7f689cc05c55ed85705c4409f8b9a42eda6f1ccd19ecfc9"
Nov 02 21:58:35.260215 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:35.260202    4528 scope.go:122] "RemoveContainer" containerID="47df24c4b6cd6beb2c204839d392407da83cc05619bceb734726e910a9730a56"
Nov 02 21:58:40.958445 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:40.958398    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:40.970481 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:40.970457    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:41.758365 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:41.758335    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 21:58:42.966245 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:42.966213    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:58:44.117978 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:58:44.117933    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:58:56.049428 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:56.049395    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"]
Nov 02 21:58:56.099438 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:56.099414    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b67e1fda-lfvvq"]
Nov 02 21:58:56.914373 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:58:56.914341    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f992866-c294-4889-a086-8b00a6fae157" path="/var/lib/kubelet/pods/0f992866-c294-4889-a086-8b00a6fae157/volumes"
Nov 02 21:59:14.113223 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:59:14.113182    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:59:34.915597 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:34.915563    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 21:59:35.328129 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:35.327931    4528 scope.go:122] "RemoveContainer" containerID="56c59d8ea38754a80aef34341016286c808221bc5800b316b73b2e3eb74f586c"
Nov 02 21:59:35.408875 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:35.408843    4528 scope.go:122] "RemoveContainer" containerID="701028591c62e641e608bc90bd4c60f78e8f005fbf4e32fe7fde03736f13e776"
Nov 02 21:59:44.113548 ip-10-0-133-86 kubenswrapper[4528]: E1102 21:59:44.113505    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 21:59:45.182358 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:45.182323    4528 generic.go:356] "Generic (PLEG): container finished" podID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerID="723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117" exitCode=0
Nov 02 21:59:45.182819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:45.182394    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" event={"ID":"d474a4c1-c212-4544-8e0c-f9e411b5b219","Type":"ContainerDied","Data":"723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117"}
Nov 02 21:59:45.182819 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:45.182751    4528 scope.go:122] "RemoveContainer" containerID="723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117"
Nov 02 21:59:45.184313 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:45.184290    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:59:45.184400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:45.184355    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:59:45.184400 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:45.184387    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 21:59:46.191454 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:46.191416    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" event={"ID":"d474a4c1-c212-4544-8e0c-f9e411b5b219","Type":"ContainerStarted","Data":"38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0"}
Nov 02 21:59:46.660347 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:46.660313    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:59:46.660601 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:46.660580    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 21:59:46.660658 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:46.660639    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 21:59:47.660863 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:47.660829    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 21:59:47.661283 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:47.660891    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 21:59:48.661506 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:48.661468    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 21:59:48.661976 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:48.661526    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 21:59:49.661523 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:49.661491    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 21:59:49.661906 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:49.661554    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 21:59:50.660684 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:50.660650    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 21:59:50.660870 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:50.660708    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 21:59:51.516147 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:51.516097    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:59:51.665176 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:51.665149    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 21:59:52.276077 ip-10-0-133-86 kubenswrapper[4528]: I1102 21:59:52.276046    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 22:00:00.181156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.181123    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"]
Nov 02 22:00:00.185728 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.185704    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"]
Nov 02 22:00:00.185877 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.185818    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:00.189647 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.189620    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"]
Nov 02 22:00:00.189754 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.189733    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:00.193855 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.193839    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:00.231857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.231834    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"]
Nov 02 22:00:00.235678 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.235663    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"]
Nov 02 22:00:00.235816 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.235795    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:00.239510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.239492    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"]
Nov 02 22:00:00.239641 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.239625    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:00.243430 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.243413    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"]
Nov 02 22:00:00.243553 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.243540    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:00.258340 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.258293    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"]
Nov 02 22:00:00.258570 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.258395    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:00.259871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.259601    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k\" (UID: \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:00.259871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.259685    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3de28a-ef2f-4efc-8611-47e5baf0a857-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9\" (UID: \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:00.259871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.259778    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpzsz\" (UniqueName: \"kubernetes.io/projected/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-kube-api-access-hpzsz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8\" (UID: \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:00.259871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.259810    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxzm\" (UniqueName: \"kubernetes.io/projected/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-kube-api-access-qnxzm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k\" (UID: \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:00.259871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.259842    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a52647f-17c7-47de-9773-2e54142b11a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz\" (UID: \"8a52647f-17c7-47de-9773-2e54142b11a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:00.260331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.259884    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqlsq\" (UniqueName: \"kubernetes.io/projected/84e3dd64-957e-4d5f-a7b1-892e58818eb3-kube-api-access-sqlsq\") pod \"maas-api-key-cleanup-29894280-g58rv\" (UID: \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:00.260331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.259921    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr\" (UID: \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:00.260331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.259954    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8\" (UID: \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:00.260331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.259990    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvdl\" (UniqueName: \"kubernetes.io/projected/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-kube-api-access-jmvdl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr\" (UID: \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:00.260331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.260036    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcg9\" (UniqueName: \"kubernetes.io/projected/3c3de28a-ef2f-4efc-8611-47e5baf0a857-kube-api-access-zkcg9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9\" (UID: \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:00.260331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.260069    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgxv\" (UniqueName: \"kubernetes.io/projected/8375e961-9f74-44ce-9569-7d596dcd5e01-kube-api-access-ftgxv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng\" (UID: \"8375e961-9f74-44ce-9569-7d596dcd5e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:00.260331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.260159    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e3dd64-957e-4d5f-a7b1-892e58818eb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894280-g58rv\" (UID: \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:00.260331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.260207    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8375e961-9f74-44ce-9569-7d596dcd5e01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng\" (UID: \"8375e961-9f74-44ce-9569-7d596dcd5e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:00.260331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.260256    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4c9\" (UniqueName: \"kubernetes.io/projected/8a52647f-17c7-47de-9773-2e54142b11a3-kube-api-access-7g4c9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz\" (UID: \"8a52647f-17c7-47de-9773-2e54142b11a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:00.274949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.272946    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"]
Nov 02 22:00:00.274949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.273032    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:00.279038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.279015    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:00:00.279666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.279650    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:00:00.279751 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.279739    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:00:00.280318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.280300    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"]
Nov 02 22:00:00.280430 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.280410    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:00.284486 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.284467    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"]
Nov 02 22:00:00.284596 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.284580    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:00.297146 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.289460    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"]
Nov 02 22:00:00.297146 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.293997    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"]
Nov 02 22:00:00.305103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.304993    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"]
Nov 02 22:00:00.310193 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.309340    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"]
Nov 02 22:00:00.314916 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.314895    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"]
Nov 02 22:00:00.334907 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.334850    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"]
Nov 02 22:00:00.347075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.347003    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"]
Nov 02 22:00:00.348084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.348060    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"]
Nov 02 22:00:00.361190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361168    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxzm\" (UniqueName: \"kubernetes.io/projected/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-kube-api-access-qnxzm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k\" (UID: \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:00.361286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361203    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a52647f-17c7-47de-9773-2e54142b11a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz\" (UID: \"8a52647f-17c7-47de-9773-2e54142b11a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:00.361286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361231    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqlsq\" (UniqueName: \"kubernetes.io/projected/84e3dd64-957e-4d5f-a7b1-892e58818eb3-kube-api-access-sqlsq\") pod \"maas-api-key-cleanup-29894280-g58rv\" (UID: \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:00.361286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361255    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25\" (UID: \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:00.361286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361282    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr\" (UID: \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:00.361513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361299    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j\" (UID: \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:00.361513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361317    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8\" (UID: \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:00.361513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361341    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x79tq\" (UniqueName: \"kubernetes.io/projected/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-kube-api-access-x79tq\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j\" (UID: \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:00.361513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361376    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvdl\" (UniqueName: \"kubernetes.io/projected/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-kube-api-access-jmvdl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr\" (UID: \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:00.361513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361418    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcg9\" (UniqueName: \"kubernetes.io/projected/3c3de28a-ef2f-4efc-8611-47e5baf0a857-kube-api-access-zkcg9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9\" (UID: \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:00.361513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361449    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgxv\" (UniqueName: \"kubernetes.io/projected/8375e961-9f74-44ce-9569-7d596dcd5e01-kube-api-access-ftgxv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng\" (UID: \"8375e961-9f74-44ce-9569-7d596dcd5e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:00.361513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361486    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e3dd64-957e-4d5f-a7b1-892e58818eb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894280-g58rv\" (UID: \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:00.361908 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361532    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8375e961-9f74-44ce-9569-7d596dcd5e01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng\" (UID: \"8375e961-9f74-44ce-9569-7d596dcd5e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:00.361908 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361575    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4c9\" (UniqueName: \"kubernetes.io/projected/8a52647f-17c7-47de-9773-2e54142b11a3-kube-api-access-7g4c9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz\" (UID: \"8a52647f-17c7-47de-9773-2e54142b11a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:00.361908 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361624    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f38da4-e241-4598-92bd-95c9085e9cdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb\" (UID: \"a3f38da4-e241-4598-92bd-95c9085e9cdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:00.361908 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361658    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k\" (UID: \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:00.361908 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361703    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg58k\" (UniqueName: \"kubernetes.io/projected/a3f38da4-e241-4598-92bd-95c9085e9cdc-kube-api-access-bg58k\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb\" (UID: \"a3f38da4-e241-4598-92bd-95c9085e9cdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:00.362204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361966    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a52647f-17c7-47de-9773-2e54142b11a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz\" (UID: \"8a52647f-17c7-47de-9773-2e54142b11a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:00.362204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.362103    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8\" (UID: \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:00.362204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.361970    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3de28a-ef2f-4efc-8611-47e5baf0a857-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9\" (UID: \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:00.362204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.362187    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqm8\" (UniqueName: \"kubernetes.io/projected/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-kube-api-access-dhqm8\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25\" (UID: \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:00.362415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.362278    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr\" (UID: \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:00.362415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.362312    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpzsz\" (UniqueName: \"kubernetes.io/projected/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-kube-api-access-hpzsz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8\" (UID: \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:00.362688 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.362656    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3de28a-ef2f-4efc-8611-47e5baf0a857-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9\" (UID: \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:00.362773 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.362672    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e3dd64-957e-4d5f-a7b1-892e58818eb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894280-g58rv\" (UID: \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:00.362983 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.362959    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8375e961-9f74-44ce-9569-7d596dcd5e01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng\" (UID: \"8375e961-9f74-44ce-9569-7d596dcd5e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:00.363277 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.363260    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k\" (UID: \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:00.374295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.374269    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"]
Nov 02 22:00:00.419450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.419410    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvdl\" (UniqueName: \"kubernetes.io/projected/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-kube-api-access-jmvdl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr\" (UID: \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:00.419599 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.419553    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqlsq\" (UniqueName: \"kubernetes.io/projected/84e3dd64-957e-4d5f-a7b1-892e58818eb3-kube-api-access-sqlsq\") pod \"maas-api-key-cleanup-29894280-g58rv\" (UID: \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:00.419677 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.419622    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcg9\" (UniqueName: \"kubernetes.io/projected/3c3de28a-ef2f-4efc-8611-47e5baf0a857-kube-api-access-zkcg9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9\" (UID: \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:00.420295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.420260    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgxv\" (UniqueName: \"kubernetes.io/projected/8375e961-9f74-44ce-9569-7d596dcd5e01-kube-api-access-ftgxv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng\" (UID: \"8375e961-9f74-44ce-9569-7d596dcd5e01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:00.420295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.420270    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxzm\" (UniqueName: \"kubernetes.io/projected/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-kube-api-access-qnxzm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k\" (UID: \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:00.420295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.420282    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4c9\" (UniqueName: \"kubernetes.io/projected/8a52647f-17c7-47de-9773-2e54142b11a3-kube-api-access-7g4c9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz\" (UID: \"8a52647f-17c7-47de-9773-2e54142b11a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:00.420556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.420534    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpzsz\" (UniqueName: \"kubernetes.io/projected/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-kube-api-access-hpzsz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8\" (UID: \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:00.463373 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.463317    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg58k\" (UniqueName: \"kubernetes.io/projected/a3f38da4-e241-4598-92bd-95c9085e9cdc-kube-api-access-bg58k\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb\" (UID: \"a3f38da4-e241-4598-92bd-95c9085e9cdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:00.463506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.463378    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqm8\" (UniqueName: \"kubernetes.io/projected/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-kube-api-access-dhqm8\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25\" (UID: \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:00.463506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.463481    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25\" (UID: \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:00.463506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.463504    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j\" (UID: \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:00.463727 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.463524    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x79tq\" (UniqueName: \"kubernetes.io/projected/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-kube-api-access-x79tq\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j\" (UID: \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:00.463727 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.463651    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f38da4-e241-4598-92bd-95c9085e9cdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb\" (UID: \"a3f38da4-e241-4598-92bd-95c9085e9cdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:00.464193 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.464176    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25\" (UID: \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:00.464240 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.464180    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j\" (UID: \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:00.464277 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.464247    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f38da4-e241-4598-92bd-95c9085e9cdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb\" (UID: \"a3f38da4-e241-4598-92bd-95c9085e9cdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:00.501338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.501317    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:00.509594 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.509579    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:00.515536 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.515507    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:00.520634 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.520608    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x79tq\" (UniqueName: \"kubernetes.io/projected/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-kube-api-access-x79tq\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j\" (UID: \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:00.520824 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.520801    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqm8\" (UniqueName: \"kubernetes.io/projected/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-kube-api-access-dhqm8\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25\" (UID: \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:00.522008 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.521840    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg58k\" (UniqueName: \"kubernetes.io/projected/a3f38da4-e241-4598-92bd-95c9085e9cdc-kube-api-access-bg58k\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb\" (UID: \"a3f38da4-e241-4598-92bd-95c9085e9cdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:00.559918 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.559877    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:00.570972 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.565402    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:00.600284 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.599954    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:00.608171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.608101    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:00.629383 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.621918    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:00.664194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.663944    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:00.669663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.669591    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:00.896317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.896287    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"]
Nov 02 22:00:00.903530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.903509    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"]
Nov 02 22:00:00.939658 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:00.939636    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"]
Nov 02 22:00:00.955435 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:00:00.955409    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e3dd64_957e_4d5f_a7b1_892e58818eb3.slice/crio-79a92d31e8353b1fef349c54eaac21ebd95db8843f1718028449325520a663ae WatchSource:0}: Error finding container 79a92d31e8353b1fef349c54eaac21ebd95db8843f1718028449325520a663ae: Status 404 returned error can't find the container with id 79a92d31e8353b1fef349c54eaac21ebd95db8843f1718028449325520a663ae
Nov 02 22:00:01.003748 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:00:01.003706    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91893bc4_e917_4f23_bc3f_a1ae147a6e1b.slice/crio-5fa15e70e610bde1c793490dfa217b5cf27a3a31dc64f5142cdcbd4170f8b1a3 WatchSource:0}: Error finding container 5fa15e70e610bde1c793490dfa217b5cf27a3a31dc64f5142cdcbd4170f8b1a3: Status 404 returned error can't find the container with id 5fa15e70e610bde1c793490dfa217b5cf27a3a31dc64f5142cdcbd4170f8b1a3
Nov 02 22:00:01.007304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.007279    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"]
Nov 02 22:00:01.034631 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.034604    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"]
Nov 02 22:00:01.064610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.064591    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"]
Nov 02 22:00:01.067570 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.067547    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"]
Nov 02 22:00:01.070186 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.070169    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"]
Nov 02 22:00:01.105374 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:00:01.105352    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a52647f_17c7_47de_9773_2e54142b11a3.slice/crio-3d1ec34e9415ec45a9b9fe3a66a6307f43df4d0a765d0026851b3d010737db31 WatchSource:0}: Error finding container 3d1ec34e9415ec45a9b9fe3a66a6307f43df4d0a765d0026851b3d010737db31: Status 404 returned error can't find the container with id 3d1ec34e9415ec45a9b9fe3a66a6307f43df4d0a765d0026851b3d010737db31
Nov 02 22:00:01.107445 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:00:01.107414    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7eaa8e_01e8_481f_9523_49b9bcc2a987.slice/crio-170b2d96453ac28e67ce3a8215270c495743dfe4dca3039bd51f81f85be5b51b WatchSource:0}: Error finding container 170b2d96453ac28e67ce3a8215270c495743dfe4dca3039bd51f81f85be5b51b: Status 404 returned error can't find the container with id 170b2d96453ac28e67ce3a8215270c495743dfe4dca3039bd51f81f85be5b51b
Nov 02 22:00:01.108391 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:00:01.108279    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8375e961_9f74_44ce_9569_7d596dcd5e01.slice/crio-f8ea6937e6b185f7ce9af8468864f14241ded7a2eaa0250af8016ecac6af6204 WatchSource:0}: Error finding container f8ea6937e6b185f7ce9af8468864f14241ded7a2eaa0250af8016ecac6af6204: Status 404 returned error can't find the container with id f8ea6937e6b185f7ce9af8468864f14241ded7a2eaa0250af8016ecac6af6204
Nov 02 22:00:01.109950 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:00:01.109843    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f38da4_e241_4598_92bd_95c9085e9cdc.slice/crio-708c3a00bbe9372c743862424a24ed33f40da253b8bb05bfd62ac333b5362a4a WatchSource:0}: Error finding container 708c3a00bbe9372c743862424a24ed33f40da253b8bb05bfd62ac333b5362a4a: Status 404 returned error can't find the container with id 708c3a00bbe9372c743862424a24ed33f40da253b8bb05bfd62ac333b5362a4a
Nov 02 22:00:01.144429 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.144179    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"]
Nov 02 22:00:01.148094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.147779    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"]
Nov 02 22:00:01.156253 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:00:01.156226    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ba9ae2_aa81_4d1d_90db_6d9890c513c3.slice/crio-6af45bdb775eaffa124f8e7d0abc6b3ee88fca182b7ae1eb02fdf6728ad1589e WatchSource:0}: Error finding container 6af45bdb775eaffa124f8e7d0abc6b3ee88fca182b7ae1eb02fdf6728ad1589e: Status 404 returned error can't find the container with id 6af45bdb775eaffa124f8e7d0abc6b3ee88fca182b7ae1eb02fdf6728ad1589e
Nov 02 22:00:01.346809 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.346780    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" event={"ID":"8a52647f-17c7-47de-9773-2e54142b11a3","Type":"ContainerStarted","Data":"3d1ec34e9415ec45a9b9fe3a66a6307f43df4d0a765d0026851b3d010737db31"}
Nov 02 22:00:01.348152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.348100    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" event={"ID":"a3f38da4-e241-4598-92bd-95c9085e9cdc","Type":"ContainerStarted","Data":"708c3a00bbe9372c743862424a24ed33f40da253b8bb05bfd62ac333b5362a4a"}
Nov 02 22:00:01.349344 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.349317    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" event={"ID":"3c3de28a-ef2f-4efc-8611-47e5baf0a857","Type":"ContainerStarted","Data":"63e974e7c42780e7400e3ee820ba1e5f64cf0faca4eb9c854626ef50dabc0c98"}
Nov 02 22:00:01.350506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.350486    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" event={"ID":"26ba9ae2-aa81-4d1d-90db-6d9890c513c3","Type":"ContainerStarted","Data":"6af45bdb775eaffa124f8e7d0abc6b3ee88fca182b7ae1eb02fdf6728ad1589e"}
Nov 02 22:00:01.351670 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.351649    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" event={"ID":"84e3dd64-957e-4d5f-a7b1-892e58818eb3","Type":"ContainerStarted","Data":"79a92d31e8353b1fef349c54eaac21ebd95db8843f1718028449325520a663ae"}
Nov 02 22:00:01.352857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.352831    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" event={"ID":"8375e961-9f74-44ce-9569-7d596dcd5e01","Type":"ContainerStarted","Data":"f8ea6937e6b185f7ce9af8468864f14241ded7a2eaa0250af8016ecac6af6204"}
Nov 02 22:00:01.353915 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.353893    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" event={"ID":"ab7eaa8e-01e8-481f-9523-49b9bcc2a987","Type":"ContainerStarted","Data":"170b2d96453ac28e67ce3a8215270c495743dfe4dca3039bd51f81f85be5b51b"}
Nov 02 22:00:01.355002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.354984    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" event={"ID":"91893bc4-e917-4f23-bc3f-a1ae147a6e1b","Type":"ContainerStarted","Data":"5fa15e70e610bde1c793490dfa217b5cf27a3a31dc64f5142cdcbd4170f8b1a3"}
Nov 02 22:00:01.356070 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.356052    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" event={"ID":"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d","Type":"ContainerStarted","Data":"51cba3052a53869a546248a33c258b9fec252827fdce6678d923f238cc0386ce"}
Nov 02 22:00:01.357202 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:01.357184    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" event={"ID":"769fb8a4-79d9-49e4-bd13-47a4d114dd0c","Type":"ContainerStarted","Data":"e028e868699baacb17b338d8ae72b2c94c481330c083462fb831d38d5658ad77"}
Nov 02 22:00:02.365877 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.365849    4528 generic.go:356] "Generic (PLEG): container finished" podID="8a52647f-17c7-47de-9773-2e54142b11a3" containerID="9b63232d4dbd9e30e1156e9f6db9a291d3fec2f35d44b61cbd21a98b880ce3ca" exitCode=60
Nov 02 22:00:02.366219 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.365929    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" event={"ID":"8a52647f-17c7-47de-9773-2e54142b11a3","Type":"ContainerDied","Data":"9b63232d4dbd9e30e1156e9f6db9a291d3fec2f35d44b61cbd21a98b880ce3ca"}
Nov 02 22:00:02.366268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.366244    4528 scope.go:122] "RemoveContainer" containerID="9b63232d4dbd9e30e1156e9f6db9a291d3fec2f35d44b61cbd21a98b880ce3ca"
Nov 02 22:00:02.367567 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.367547    4528 generic.go:356] "Generic (PLEG): container finished" podID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerID="0bf53a23d3d0657d4ad4de1560956c8b4b55d100feed42118c17cae4dea53dfd" exitCode=60
Nov 02 22:00:02.367668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.367620    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" event={"ID":"a3f38da4-e241-4598-92bd-95c9085e9cdc","Type":"ContainerDied","Data":"0bf53a23d3d0657d4ad4de1560956c8b4b55d100feed42118c17cae4dea53dfd"}
Nov 02 22:00:02.367896 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.367844    4528 scope.go:122] "RemoveContainer" containerID="0bf53a23d3d0657d4ad4de1560956c8b4b55d100feed42118c17cae4dea53dfd"
Nov 02 22:00:02.369173 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.369153    4528 generic.go:356] "Generic (PLEG): container finished" podID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerID="9896cebf3588832c102f594cce09f04955edfb9aea98dc78d6828e1986e4ebd3" exitCode=60
Nov 02 22:00:02.369277 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.369226    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" event={"ID":"3c3de28a-ef2f-4efc-8611-47e5baf0a857","Type":"ContainerDied","Data":"9896cebf3588832c102f594cce09f04955edfb9aea98dc78d6828e1986e4ebd3"}
Nov 02 22:00:02.369417 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.369402    4528 scope.go:122] "RemoveContainer" containerID="9896cebf3588832c102f594cce09f04955edfb9aea98dc78d6828e1986e4ebd3"
Nov 02 22:00:02.371038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.370982    4528 generic.go:356] "Generic (PLEG): container finished" podID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerID="f530b19b82dfc398699c41d3e483ddbd9302051833708ab45d00ef659ceb760d" exitCode=60
Nov 02 22:00:02.371103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.371061    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" event={"ID":"26ba9ae2-aa81-4d1d-90db-6d9890c513c3","Type":"ContainerDied","Data":"f530b19b82dfc398699c41d3e483ddbd9302051833708ab45d00ef659ceb760d"}
Nov 02 22:00:02.371300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.371283    4528 scope.go:122] "RemoveContainer" containerID="f530b19b82dfc398699c41d3e483ddbd9302051833708ab45d00ef659ceb760d"
Nov 02 22:00:02.372718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.372699    4528 generic.go:356] "Generic (PLEG): container finished" podID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerID="646b8b76196bd4f5f5468c3f9cb26df3b5f25a404ac0b89844ed08184c37fa45" exitCode=60
Nov 02 22:00:02.372821 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.372777    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" event={"ID":"84e3dd64-957e-4d5f-a7b1-892e58818eb3","Type":"ContainerDied","Data":"646b8b76196bd4f5f5468c3f9cb26df3b5f25a404ac0b89844ed08184c37fa45"}
Nov 02 22:00:02.372945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.372929    4528 scope.go:122] "RemoveContainer" containerID="646b8b76196bd4f5f5468c3f9cb26df3b5f25a404ac0b89844ed08184c37fa45"
Nov 02 22:00:02.374354 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.374332    4528 generic.go:356] "Generic (PLEG): container finished" podID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerID="d6365c01227b81af9fd667917ae6516f57f2d7be6172808281bcf3596ef1edcb" exitCode=60
Nov 02 22:00:02.374457 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.374410    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" event={"ID":"8375e961-9f74-44ce-9569-7d596dcd5e01","Type":"ContainerDied","Data":"d6365c01227b81af9fd667917ae6516f57f2d7be6172808281bcf3596ef1edcb"}
Nov 02 22:00:02.374563 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.374549    4528 scope.go:122] "RemoveContainer" containerID="d6365c01227b81af9fd667917ae6516f57f2d7be6172808281bcf3596ef1edcb"
Nov 02 22:00:02.375959 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.375941    4528 generic.go:356] "Generic (PLEG): container finished" podID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerID="370107c90d3c45f5d3598c3a5d1d00b34c400b9aa9705de8f633b16bbe824a41" exitCode=60
Nov 02 22:00:02.376059 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.375996    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" event={"ID":"ab7eaa8e-01e8-481f-9523-49b9bcc2a987","Type":"ContainerDied","Data":"370107c90d3c45f5d3598c3a5d1d00b34c400b9aa9705de8f633b16bbe824a41"}
Nov 02 22:00:02.376221 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.376192    4528 scope.go:122] "RemoveContainer" containerID="370107c90d3c45f5d3598c3a5d1d00b34c400b9aa9705de8f633b16bbe824a41"
Nov 02 22:00:02.378338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.377560    4528 generic.go:356] "Generic (PLEG): container finished" podID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerID="2a68d1b46fed63d1cba19872425cb974b4dc626d9db104a57614f7503416def9" exitCode=60
Nov 02 22:00:02.378338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.377896    4528 scope.go:122] "RemoveContainer" containerID="2a68d1b46fed63d1cba19872425cb974b4dc626d9db104a57614f7503416def9"
Nov 02 22:00:02.378338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.378058    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" event={"ID":"91893bc4-e917-4f23-bc3f-a1ae147a6e1b","Type":"ContainerDied","Data":"2a68d1b46fed63d1cba19872425cb974b4dc626d9db104a57614f7503416def9"}
Nov 02 22:00:02.380379 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.380351    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" event={"ID":"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d","Type":"ContainerStarted","Data":"c242b384d2e54b49c492561270eeb6dca466f8fcf6c1c11bbf15d7c68ea769cb"}
Nov 02 22:00:02.381862 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.381841    4528 generic.go:356] "Generic (PLEG): container finished" podID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerID="8472b2564878feef9fc8b0eb8d3d763d29dedc204fec8a01a2b9a2e5cdba6368" exitCode=60
Nov 02 22:00:02.381951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.381920    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" event={"ID":"769fb8a4-79d9-49e4-bd13-47a4d114dd0c","Type":"ContainerDied","Data":"8472b2564878feef9fc8b0eb8d3d763d29dedc204fec8a01a2b9a2e5cdba6368"}
Nov 02 22:00:02.382122 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:02.382095    4528 scope.go:122] "RemoveContainer" containerID="8472b2564878feef9fc8b0eb8d3d763d29dedc204fec8a01a2b9a2e5cdba6368"
Nov 02 22:00:03.390695 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.390665    4528 generic.go:356] "Generic (PLEG): container finished" podID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerID="c242b384d2e54b49c492561270eeb6dca466f8fcf6c1c11bbf15d7c68ea769cb" exitCode=60
Nov 02 22:00:03.391149 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.390710    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" event={"ID":"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d","Type":"ContainerDied","Data":"c242b384d2e54b49c492561270eeb6dca466f8fcf6c1c11bbf15d7c68ea769cb"}
Nov 02 22:00:03.391149 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.391038    4528 scope.go:122] "RemoveContainer" containerID="c242b384d2e54b49c492561270eeb6dca466f8fcf6c1c11bbf15d7c68ea769cb"
Nov 02 22:00:03.392964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.392868    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" event={"ID":"769fb8a4-79d9-49e4-bd13-47a4d114dd0c","Type":"ContainerStarted","Data":"3b792e2b9161606be9df284b9a5024c8f81c3132bb3b769af503c104676076ed"}
Nov 02 22:00:03.394767 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.394741    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" event={"ID":"8a52647f-17c7-47de-9773-2e54142b11a3","Type":"ContainerStarted","Data":"bd36e05bcb43bcc395893d456087069bf4dbe712c9d1f99867417bcf602d7d0c"}
Nov 02 22:00:03.397428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.397410    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" event={"ID":"a3f38da4-e241-4598-92bd-95c9085e9cdc","Type":"ContainerStarted","Data":"b5aadc338a499d8701e31f5ecf23433517d46e1c09b7e1c86afda13ad10b2560"}
Nov 02 22:00:03.399467 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.399448    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" event={"ID":"3c3de28a-ef2f-4efc-8611-47e5baf0a857","Type":"ContainerStarted","Data":"abfc7ea7941d77e8bccbc02f10b622a539d9cc5ed27778aff78563af6bd3aa6a"}
Nov 02 22:00:03.404751 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.401372    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" event={"ID":"26ba9ae2-aa81-4d1d-90db-6d9890c513c3","Type":"ContainerStarted","Data":"00f0e9be56132218fdbf9bbf02ed959e4a8b8b42db589a3f8be2ebdc7da99a17"}
Nov 02 22:00:03.404751 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.403837    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" event={"ID":"84e3dd64-957e-4d5f-a7b1-892e58818eb3","Type":"ContainerStarted","Data":"6f278f1efa6a8a8d57a1d07e227b48cc07ee66e98802d65e91a5fa82f70e6dfd"}
Nov 02 22:00:03.406647 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.406625    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" event={"ID":"8375e961-9f74-44ce-9569-7d596dcd5e01","Type":"ContainerStarted","Data":"193817d4f7d614f7af7c0000c733dff28e8c832ae9d791b804de621f31121956"}
Nov 02 22:00:03.408375 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.408348    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" event={"ID":"ab7eaa8e-01e8-481f-9523-49b9bcc2a987","Type":"ContainerStarted","Data":"3ee14b016f7902ac65d89bdaaf02eb95430cba4bf7ffdfa90740c6f4ae986b95"}
Nov 02 22:00:03.410201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.410182    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" event={"ID":"91893bc4-e917-4f23-bc3f-a1ae147a6e1b","Type":"ContainerStarted","Data":"19f1315d2775719040974043b420053b83e19ba898929a2fc029ffea49e85bc5"}
Nov 02 22:00:03.418861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.418821    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" podStartSLOduration=3.418810311 podStartE2EDuration="3.418810311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.417103644 +0000 UTC m=+3090.680639961" watchObservedRunningTime="2026-11-02 22:00:03.418810311 +0000 UTC m=+3090.682346668"
Nov 02 22:00:03.642626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.642575    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" podStartSLOduration=3.642560531 podStartE2EDuration="3.642560531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.642199135 +0000 UTC m=+3090.905735455" watchObservedRunningTime="2026-11-02 22:00:03.642560531 +0000 UTC m=+3090.906096838"
Nov 02 22:00:03.755649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.755599    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" podStartSLOduration=3.755584827 podStartE2EDuration="3.755584827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.754099714 +0000 UTC m=+3091.017636033" watchObservedRunningTime="2026-11-02 22:00:03.755584827 +0000 UTC m=+3091.019121144"
Nov 02 22:00:03.868648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.868599    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" podStartSLOduration=3.868584342 podStartE2EDuration="3.868584342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.866311906 +0000 UTC m=+3091.129848222" watchObservedRunningTime="2026-11-02 22:00:03.868584342 +0000 UTC m=+3091.132120657"
Nov 02 22:00:03.983600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:03.983554    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" podStartSLOduration=3.9835402479999997 podStartE2EDuration="3.983540248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.979744173 +0000 UTC m=+3091.243280521" watchObservedRunningTime="2026-11-02 22:00:03.983540248 +0000 UTC m=+3091.247076565"
Nov 02 22:00:04.094732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.094653    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" podStartSLOduration=4.094639057 podStartE2EDuration="4.094639057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:04.09241505 +0000 UTC m=+3091.355951366" watchObservedRunningTime="2026-11-02 22:00:04.094639057 +0000 UTC m=+3091.358175373"
Nov 02 22:00:04.319053 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.318999    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" podStartSLOduration=4.318984091 podStartE2EDuration="4.318984091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:04.31697905 +0000 UTC m=+3091.580515366" watchObservedRunningTime="2026-11-02 22:00:04.318984091 +0000 UTC m=+3091.582520407"
Nov 02 22:00:04.418852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.418771    4528 generic.go:356] "Generic (PLEG): container finished" podID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerID="00f0e9be56132218fdbf9bbf02ed959e4a8b8b42db589a3f8be2ebdc7da99a17" exitCode=60
Nov 02 22:00:04.419318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.418852    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" event={"ID":"26ba9ae2-aa81-4d1d-90db-6d9890c513c3","Type":"ContainerDied","Data":"00f0e9be56132218fdbf9bbf02ed959e4a8b8b42db589a3f8be2ebdc7da99a17"}
Nov 02 22:00:04.419318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.418948    4528 scope.go:122] "RemoveContainer" containerID="f530b19b82dfc398699c41d3e483ddbd9302051833708ab45d00ef659ceb760d"
Nov 02 22:00:04.419318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.419140    4528 scope.go:122] "RemoveContainer" containerID="00f0e9be56132218fdbf9bbf02ed959e4a8b8b42db589a3f8be2ebdc7da99a17"
Nov 02 22:00:04.419520 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:04.419499    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j_odh-ai-gateway-infra(26ba9ae2-aa81-4d1d-90db-6d9890c513c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3"
Nov 02 22:00:04.420824 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.420801    4528 generic.go:356] "Generic (PLEG): container finished" podID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerID="6f278f1efa6a8a8d57a1d07e227b48cc07ee66e98802d65e91a5fa82f70e6dfd" exitCode=60
Nov 02 22:00:04.420947 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.420863    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" event={"ID":"84e3dd64-957e-4d5f-a7b1-892e58818eb3","Type":"ContainerDied","Data":"6f278f1efa6a8a8d57a1d07e227b48cc07ee66e98802d65e91a5fa82f70e6dfd"}
Nov 02 22:00:04.421102 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.421089    4528 scope.go:122] "RemoveContainer" containerID="6f278f1efa6a8a8d57a1d07e227b48cc07ee66e98802d65e91a5fa82f70e6dfd"
Nov 02 22:00:04.421406 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:04.421387    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894280-g58rv_odh-ai-gateway-infra(84e3dd64-957e-4d5f-a7b1-892e58818eb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3"
Nov 02 22:00:04.422635 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.422617    4528 generic.go:356] "Generic (PLEG): container finished" podID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerID="193817d4f7d614f7af7c0000c733dff28e8c832ae9d791b804de621f31121956" exitCode=60
Nov 02 22:00:04.422743 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.422691    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" event={"ID":"8375e961-9f74-44ce-9569-7d596dcd5e01","Type":"ContainerDied","Data":"193817d4f7d614f7af7c0000c733dff28e8c832ae9d791b804de621f31121956"}
Nov 02 22:00:04.422945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.422932    4528 scope.go:122] "RemoveContainer" containerID="193817d4f7d614f7af7c0000c733dff28e8c832ae9d791b804de621f31121956"
Nov 02 22:00:04.423247 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:04.423225    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng_odh-ai-gateway-infra(8375e961-9f74-44ce-9569-7d596dcd5e01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01"
Nov 02 22:00:04.424709 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.424689    4528 generic.go:356] "Generic (PLEG): container finished" podID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerID="3ee14b016f7902ac65d89bdaaf02eb95430cba4bf7ffdfa90740c6f4ae986b95" exitCode=60
Nov 02 22:00:04.424802 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.424758    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" event={"ID":"ab7eaa8e-01e8-481f-9523-49b9bcc2a987","Type":"ContainerDied","Data":"3ee14b016f7902ac65d89bdaaf02eb95430cba4bf7ffdfa90740c6f4ae986b95"}
Nov 02 22:00:04.424990 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.424974    4528 scope.go:122] "RemoveContainer" containerID="3ee14b016f7902ac65d89bdaaf02eb95430cba4bf7ffdfa90740c6f4ae986b95"
Nov 02 22:00:04.425255 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:04.425236    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8_odh-ai-gateway-infra(ab7eaa8e-01e8-481f-9523-49b9bcc2a987)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987"
Nov 02 22:00:04.426759 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.426739    4528 generic.go:356] "Generic (PLEG): container finished" podID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerID="19f1315d2775719040974043b420053b83e19ba898929a2fc029ffea49e85bc5" exitCode=60
Nov 02 22:00:04.426881 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.426819    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" event={"ID":"91893bc4-e917-4f23-bc3f-a1ae147a6e1b","Type":"ContainerDied","Data":"19f1315d2775719040974043b420053b83e19ba898929a2fc029ffea49e85bc5"}
Nov 02 22:00:04.427057 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.427041    4528 scope.go:122] "RemoveContainer" containerID="19f1315d2775719040974043b420053b83e19ba898929a2fc029ffea49e85bc5"
Nov 02 22:00:04.427499 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:04.427477    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k_odh-ai-gateway-infra(91893bc4-e917-4f23-bc3f-a1ae147a6e1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b"
Nov 02 22:00:04.428832 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.428810    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" event={"ID":"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d","Type":"ContainerStarted","Data":"ec7d63ca4500ce97127c7ad8930a20ea71d0276b0519449552d03c4d818a23b0"}
Nov 02 22:00:04.435240 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.435219    4528 scope.go:122] "RemoveContainer" containerID="646b8b76196bd4f5f5468c3f9cb26df3b5f25a404ac0b89844ed08184c37fa45"
Nov 02 22:00:04.437073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.437035    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" podStartSLOduration=4.437023297 podStartE2EDuration="4.437023297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:04.434987526 +0000 UTC m=+3091.698523844" watchObservedRunningTime="2026-11-02 22:00:04.437023297 +0000 UTC m=+3091.700559613"
Nov 02 22:00:04.437218 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.437196    4528 generic.go:356] "Generic (PLEG): container finished" podID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerID="3b792e2b9161606be9df284b9a5024c8f81c3132bb3b769af503c104676076ed" exitCode=60
Nov 02 22:00:04.437271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.437221    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" event={"ID":"769fb8a4-79d9-49e4-bd13-47a4d114dd0c","Type":"ContainerDied","Data":"3b792e2b9161606be9df284b9a5024c8f81c3132bb3b769af503c104676076ed"}
Nov 02 22:00:04.437501 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.437475    4528 scope.go:122] "RemoveContainer" containerID="3b792e2b9161606be9df284b9a5024c8f81c3132bb3b769af503c104676076ed"
Nov 02 22:00:04.437734 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:04.437713    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr_odh-ai-gateway-infra(769fb8a4-79d9-49e4-bd13-47a4d114dd0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c"
Nov 02 22:00:04.439094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.439075    4528 generic.go:356] "Generic (PLEG): container finished" podID="8a52647f-17c7-47de-9773-2e54142b11a3" containerID="bd36e05bcb43bcc395893d456087069bf4dbe712c9d1f99867417bcf602d7d0c" exitCode=60
Nov 02 22:00:04.439221 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.439158    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" event={"ID":"8a52647f-17c7-47de-9773-2e54142b11a3","Type":"ContainerDied","Data":"bd36e05bcb43bcc395893d456087069bf4dbe712c9d1f99867417bcf602d7d0c"}
Nov 02 22:00:04.439368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.439354    4528 scope.go:122] "RemoveContainer" containerID="bd36e05bcb43bcc395893d456087069bf4dbe712c9d1f99867417bcf602d7d0c"
Nov 02 22:00:04.439601 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:04.439576    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz_odh-ai-gateway-infra(8a52647f-17c7-47de-9773-2e54142b11a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" podUID="8a52647f-17c7-47de-9773-2e54142b11a3"
Nov 02 22:00:04.440994 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.440970    4528 generic.go:356] "Generic (PLEG): container finished" podID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerID="b5aadc338a499d8701e31f5ecf23433517d46e1c09b7e1c86afda13ad10b2560" exitCode=60
Nov 02 22:00:04.441103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.441045    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" event={"ID":"a3f38da4-e241-4598-92bd-95c9085e9cdc","Type":"ContainerDied","Data":"b5aadc338a499d8701e31f5ecf23433517d46e1c09b7e1c86afda13ad10b2560"}
Nov 02 22:00:04.441276 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.441260    4528 scope.go:122] "RemoveContainer" containerID="b5aadc338a499d8701e31f5ecf23433517d46e1c09b7e1c86afda13ad10b2560"
Nov 02 22:00:04.441453 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:04.441432    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb_odh-ai-gateway-infra(a3f38da4-e241-4598-92bd-95c9085e9cdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc"
Nov 02 22:00:04.442731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.442708    4528 generic.go:356] "Generic (PLEG): container finished" podID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerID="abfc7ea7941d77e8bccbc02f10b622a539d9cc5ed27778aff78563af6bd3aa6a" exitCode=60
Nov 02 22:00:04.442819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.442743    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" event={"ID":"3c3de28a-ef2f-4efc-8611-47e5baf0a857","Type":"ContainerDied","Data":"abfc7ea7941d77e8bccbc02f10b622a539d9cc5ed27778aff78563af6bd3aa6a"}
Nov 02 22:00:04.442951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.442937    4528 scope.go:122] "RemoveContainer" containerID="abfc7ea7941d77e8bccbc02f10b622a539d9cc5ed27778aff78563af6bd3aa6a"
Nov 02 22:00:04.443146 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:04.443099    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9_odh-ai-gateway-infra(3c3de28a-ef2f-4efc-8611-47e5baf0a857)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857"
Nov 02 22:00:04.448481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.448464    4528 scope.go:122] "RemoveContainer" containerID="d6365c01227b81af9fd667917ae6516f57f2d7be6172808281bcf3596ef1edcb"
Nov 02 22:00:04.460793 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.460774    4528 scope.go:122] "RemoveContainer" containerID="370107c90d3c45f5d3598c3a5d1d00b34c400b9aa9705de8f633b16bbe824a41"
Nov 02 22:00:04.472792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.472776    4528 scope.go:122] "RemoveContainer" containerID="2a68d1b46fed63d1cba19872425cb974b4dc626d9db104a57614f7503416def9"
Nov 02 22:00:04.484609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.484593    4528 scope.go:122] "RemoveContainer" containerID="8472b2564878feef9fc8b0eb8d3d763d29dedc204fec8a01a2b9a2e5cdba6368"
Nov 02 22:00:04.496879 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.496865    4528 scope.go:122] "RemoveContainer" containerID="9b63232d4dbd9e30e1156e9f6db9a291d3fec2f35d44b61cbd21a98b880ce3ca"
Nov 02 22:00:04.548961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.548923    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" podStartSLOduration=4.548911851 podStartE2EDuration="4.548911851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:04.547658867 +0000 UTC m=+3091.811195186" watchObservedRunningTime="2026-11-02 22:00:04.548911851 +0000 UTC m=+3091.812448165"
Nov 02 22:00:04.564488 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.564466    4528 scope.go:122] "RemoveContainer" containerID="0bf53a23d3d0657d4ad4de1560956c8b4b55d100feed42118c17cae4dea53dfd"
Nov 02 22:00:04.576821 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.576803    4528 scope.go:122] "RemoveContainer" containerID="9896cebf3588832c102f594cce09f04955edfb9aea98dc78d6828e1986e4ebd3"
Nov 02 22:00:04.666207 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:04.666155    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" podStartSLOduration=4.6661382719999995 podStartE2EDuration="4.666138272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:04.660697935 +0000 UTC m=+3091.924234276" watchObservedRunningTime="2026-11-02 22:00:04.666138272 +0000 UTC m=+3091.929674590"
Nov 02 22:00:05.453718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.453687    4528 scope.go:122] "RemoveContainer" containerID="6f278f1efa6a8a8d57a1d07e227b48cc07ee66e98802d65e91a5fa82f70e6dfd"
Nov 02 22:00:05.454190 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.453975    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894280-g58rv_odh-ai-gateway-infra(84e3dd64-957e-4d5f-a7b1-892e58818eb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3"
Nov 02 22:00:05.455664 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.455638    4528 scope.go:122] "RemoveContainer" containerID="193817d4f7d614f7af7c0000c733dff28e8c832ae9d791b804de621f31121956"
Nov 02 22:00:05.455894 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.455860    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng_odh-ai-gateway-infra(8375e961-9f74-44ce-9569-7d596dcd5e01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01"
Nov 02 22:00:05.457269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.457249    4528 scope.go:122] "RemoveContainer" containerID="3ee14b016f7902ac65d89bdaaf02eb95430cba4bf7ffdfa90740c6f4ae986b95"
Nov 02 22:00:05.457499 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.457462    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8_odh-ai-gateway-infra(ab7eaa8e-01e8-481f-9523-49b9bcc2a987)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987"
Nov 02 22:00:05.458726 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.458710    4528 scope.go:122] "RemoveContainer" containerID="19f1315d2775719040974043b420053b83e19ba898929a2fc029ffea49e85bc5"
Nov 02 22:00:05.458935 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.458917    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k_odh-ai-gateway-infra(91893bc4-e917-4f23-bc3f-a1ae147a6e1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b"
Nov 02 22:00:05.460437 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.460417    4528 generic.go:356] "Generic (PLEG): container finished" podID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerID="ec7d63ca4500ce97127c7ad8930a20ea71d0276b0519449552d03c4d818a23b0" exitCode=60
Nov 02 22:00:05.460518 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.460483    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" event={"ID":"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d","Type":"ContainerDied","Data":"ec7d63ca4500ce97127c7ad8930a20ea71d0276b0519449552d03c4d818a23b0"}
Nov 02 22:00:05.460580 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.460524    4528 scope.go:122] "RemoveContainer" containerID="c242b384d2e54b49c492561270eeb6dca466f8fcf6c1c11bbf15d7c68ea769cb"
Nov 02 22:00:05.460672 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.460655    4528 scope.go:122] "RemoveContainer" containerID="ec7d63ca4500ce97127c7ad8930a20ea71d0276b0519449552d03c4d818a23b0"
Nov 02 22:00:05.460882 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.460860    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25_odh-ai-gateway-infra(fb0bfed4-b5c9-4f30-b136-87b8ad130d5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d"
Nov 02 22:00:05.462225 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.462205    4528 scope.go:122] "RemoveContainer" containerID="3b792e2b9161606be9df284b9a5024c8f81c3132bb3b769af503c104676076ed"
Nov 02 22:00:05.462430 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.462406    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr_odh-ai-gateway-infra(769fb8a4-79d9-49e4-bd13-47a4d114dd0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c"
Nov 02 22:00:05.463795 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.463779    4528 scope.go:122] "RemoveContainer" containerID="bd36e05bcb43bcc395893d456087069bf4dbe712c9d1f99867417bcf602d7d0c"
Nov 02 22:00:05.463992 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.463968    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz_odh-ai-gateway-infra(8a52647f-17c7-47de-9773-2e54142b11a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" podUID="8a52647f-17c7-47de-9773-2e54142b11a3"
Nov 02 22:00:05.465415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.465398    4528 scope.go:122] "RemoveContainer" containerID="b5aadc338a499d8701e31f5ecf23433517d46e1c09b7e1c86afda13ad10b2560"
Nov 02 22:00:05.465609 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.465591    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb_odh-ai-gateway-infra(a3f38da4-e241-4598-92bd-95c9085e9cdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc"
Nov 02 22:00:05.466794 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.466772    4528 scope.go:122] "RemoveContainer" containerID="abfc7ea7941d77e8bccbc02f10b622a539d9cc5ed27778aff78563af6bd3aa6a"
Nov 02 22:00:05.467016 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.466992    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9_odh-ai-gateway-infra(3c3de28a-ef2f-4efc-8611-47e5baf0a857)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857"
Nov 02 22:00:05.468294 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:05.468279    4528 scope.go:122] "RemoveContainer" containerID="00f0e9be56132218fdbf9bbf02ed959e4a8b8b42db589a3f8be2ebdc7da99a17"
Nov 02 22:00:05.468454 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:05.468438    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j_odh-ai-gateway-infra(26ba9ae2-aa81-4d1d-90db-6d9890c513c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3"
Nov 02 22:00:06.476338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:06.476313    4528 scope.go:122] "RemoveContainer" containerID="ec7d63ca4500ce97127c7ad8930a20ea71d0276b0519449552d03c4d818a23b0"
Nov 02 22:00:06.476697 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:06.476575    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25_odh-ai-gateway-infra(fb0bfed4-b5c9-4f30-b136-87b8ad130d5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d"
Nov 02 22:00:09.994631 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:09.994597    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"]
Nov 02 22:00:10.102005 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:10.101974    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"]
Nov 02 22:00:11.104602 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:11.104567    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:00:14.113233 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:14.113193    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:00:14.906384 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:14.906353    4528 scope.go:122] "RemoveContainer" containerID="b5aadc338a499d8701e31f5ecf23433517d46e1c09b7e1c86afda13ad10b2560"
Nov 02 22:00:14.906649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:14.906627    4528 scope.go:122] "RemoveContainer" containerID="19f1315d2775719040974043b420053b83e19ba898929a2fc029ffea49e85bc5"
Nov 02 22:00:14.906731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:14.906678    4528 scope.go:122] "RemoveContainer" containerID="193817d4f7d614f7af7c0000c733dff28e8c832ae9d791b804de621f31121956"
Nov 02 22:00:14.995463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:14.995403    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7" podUID="2c2f8896-050b-4a6b-a685-2975ac72b63f" containerName="payload-processing" containerID="cri-o://2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499" gracePeriod=25
Nov 02 22:00:15.102508 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.102474    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v" podUID="08ea82f2-85a7-4e4a-bd18-1ef77dacb533" containerName="payload-pre-processing" containerID="cri-o://22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b" gracePeriod=25
Nov 02 22:00:15.289708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.289684    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 22:00:15.311146 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.311100    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2f8896-050b-4a6b-a685-2975ac72b63f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2f8896-050b-4a6b-a685-2975ac72b63f-plugins-config-volume\") pod \"2c2f8896-050b-4a6b-a685-2975ac72b63f\" (UID: \"2c2f8896-050b-4a6b-a685-2975ac72b63f\") "
Nov 02 22:00:15.311278 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.311270    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2f8896-050b-4a6b-a685-2975ac72b63f-kube-api-access-7m98r\" (UniqueName: \"kubernetes.io/projected/2c2f8896-050b-4a6b-a685-2975ac72b63f-kube-api-access-7m98r\") pod \"2c2f8896-050b-4a6b-a685-2975ac72b63f\" (UID: \"2c2f8896-050b-4a6b-a685-2975ac72b63f\") "
Nov 02 22:00:15.311408 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.311382    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2f8896-050b-4a6b-a685-2975ac72b63f-plugins-config-volume" pod "2c2f8896-050b-4a6b-a685-2975ac72b63f" (UID: "2c2f8896-050b-4a6b-a685-2975ac72b63f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:15.311747 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.311726    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c2f8896-050b-4a6b-a685-2975ac72b63f-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:15.313263 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.313243    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2f8896-050b-4a6b-a685-2975ac72b63f-kube-api-access-7m98r" pod "2c2f8896-050b-4a6b-a685-2975ac72b63f" (UID: "2c2f8896-050b-4a6b-a685-2975ac72b63f"). InnerVolumeSpecName "kube-api-access-7m98r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:15.373707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.373683    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 22:00:15.412358 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.412333    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-plugins-config-volume\") pod \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\" (UID: \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\") "
Nov 02 22:00:15.412581 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.412565    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-kube-api-access-mmd4t\" (UniqueName: \"kubernetes.io/projected/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-kube-api-access-mmd4t\") pod \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\" (UID: \"08ea82f2-85a7-4e4a-bd18-1ef77dacb533\") "
Nov 02 22:00:15.412731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.412706    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-plugins-config-volume" pod "08ea82f2-85a7-4e4a-bd18-1ef77dacb533" (UID: "08ea82f2-85a7-4e4a-bd18-1ef77dacb533"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:15.413209 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.413184    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:15.413297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.413215    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m98r\" (UniqueName: \"kubernetes.io/projected/2c2f8896-050b-4a6b-a685-2975ac72b63f-kube-api-access-7m98r\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:15.414545 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.414523    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-kube-api-access-mmd4t" pod "08ea82f2-85a7-4e4a-bd18-1ef77dacb533" (UID: "08ea82f2-85a7-4e4a-bd18-1ef77dacb533"). InnerVolumeSpecName "kube-api-access-mmd4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:15.514037 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.514011    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmd4t\" (UniqueName: \"kubernetes.io/projected/08ea82f2-85a7-4e4a-bd18-1ef77dacb533-kube-api-access-mmd4t\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:15.595347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.595310    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" event={"ID":"8375e961-9f74-44ce-9569-7d596dcd5e01","Type":"ContainerStarted","Data":"7b104d3ebb338a59986b164007aa0ffe2fd682cc5f45cf73a79f245e869ead12"}
Nov 02 22:00:15.596782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.596750    4528 generic.go:356] "Generic (PLEG): container finished" podID="08ea82f2-85a7-4e4a-bd18-1ef77dacb533" containerID="22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b" exitCode=0
Nov 02 22:00:15.596894 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.596792    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"
Nov 02 22:00:15.596894 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.596827    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v" event={"ID":"08ea82f2-85a7-4e4a-bd18-1ef77dacb533","Type":"ContainerDied","Data":"22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b"}
Nov 02 22:00:15.596894 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.596874    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v" event={"ID":"08ea82f2-85a7-4e4a-bd18-1ef77dacb533","Type":"ContainerDied","Data":"25114381f3a4848576abe1631fb026a5fc50d9dc810cb1c0cf812e0e8892982b"}
Nov 02 22:00:15.597092 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.596897    4528 scope.go:122] "RemoveContainer" containerID="22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b"
Nov 02 22:00:15.598894 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.598870    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" event={"ID":"91893bc4-e917-4f23-bc3f-a1ae147a6e1b","Type":"ContainerStarted","Data":"eeeb0853ec744bbc4f915a7c5c3fad8db016d1c66eff5e95b994efd33916a818"}
Nov 02 22:00:15.601042 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.601022    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" event={"ID":"a3f38da4-e241-4598-92bd-95c9085e9cdc","Type":"ContainerStarted","Data":"108d07cef3c68bffe4c196dd5cb855a7d044a72146b7b1277a0039d08895b208"}
Nov 02 22:00:15.604143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.602811    4528 generic.go:356] "Generic (PLEG): container finished" podID="2c2f8896-050b-4a6b-a685-2975ac72b63f" containerID="2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499" exitCode=0
Nov 02 22:00:15.604143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.602851    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"
Nov 02 22:00:15.604143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.602889    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7" event={"ID":"2c2f8896-050b-4a6b-a685-2975ac72b63f","Type":"ContainerDied","Data":"2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499"}
Nov 02 22:00:15.604143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.602916    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7" event={"ID":"2c2f8896-050b-4a6b-a685-2975ac72b63f","Type":"ContainerDied","Data":"12d269223773b61ef62a35eea9722e997c6f831dc28bf883ec70685f990e6ea8"}
Nov 02 22:00:15.608797 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.608779    4528 scope.go:122] "RemoveContainer" containerID="22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b"
Nov 02 22:00:15.609088 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:15.609064    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b\": container with ID starting with 22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b not found: ID does not exist" containerID="22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b"
Nov 02 22:00:15.609301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.609096    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b"} err="failed to get container status \"22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b\": rpc error: code = NotFound desc = could not find container \"22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b\": container with ID starting with 22d3b62b82eee6ddb202f5718f2af83d504707148eb670b35c41ccf36f6a303b not found: ID does not exist"
Nov 02 22:00:15.609301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.609132    4528 scope.go:122] "RemoveContainer" containerID="2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499"
Nov 02 22:00:15.618681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.618659    4528 scope.go:122] "RemoveContainer" containerID="2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499"
Nov 02 22:00:15.618956 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:15.618933    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499\": container with ID starting with 2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499 not found: ID does not exist" containerID="2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499"
Nov 02 22:00:15.619017 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.618965    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499"} err="failed to get container status \"2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499\": rpc error: code = NotFound desc = could not find container \"2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499\": container with ID starting with 2f096de00ee670dee39cfd6a156cf7c25fc7409ebf2a0e222adcde887231a499 not found: ID does not exist"
Nov 02 22:00:15.906984 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.906943    4528 scope.go:122] "RemoveContainer" containerID="3ee14b016f7902ac65d89bdaaf02eb95430cba4bf7ffdfa90740c6f4ae986b95"
Nov 02 22:00:15.907175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.907095    4528 scope.go:122] "RemoveContainer" containerID="abfc7ea7941d77e8bccbc02f10b622a539d9cc5ed27778aff78563af6bd3aa6a"
Nov 02 22:00:15.907374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.907355    4528 scope.go:122] "RemoveContainer" containerID="bd36e05bcb43bcc395893d456087069bf4dbe712c9d1f99867417bcf602d7d0c"
Nov 02 22:00:15.907516 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.907499    4528 scope.go:122] "RemoveContainer" containerID="3b792e2b9161606be9df284b9a5024c8f81c3132bb3b769af503c104676076ed"
Nov 02 22:00:15.932010 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.931989    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"]
Nov 02 22:00:15.935789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:15.935764    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4ca09f-76688568bc-mbkf7"]
Nov 02 22:00:16.158750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.158686    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"]
Nov 02 22:00:16.162144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.162099    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4ca09f-68bf87579-df62v"]
Nov 02 22:00:16.612681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.612651    4528 generic.go:356] "Generic (PLEG): container finished" podID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerID="7b104d3ebb338a59986b164007aa0ffe2fd682cc5f45cf73a79f245e869ead12" exitCode=60
Nov 02 22:00:16.613191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.612720    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" event={"ID":"8375e961-9f74-44ce-9569-7d596dcd5e01","Type":"ContainerDied","Data":"7b104d3ebb338a59986b164007aa0ffe2fd682cc5f45cf73a79f245e869ead12"}
Nov 02 22:00:16.613191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.612805    4528 scope.go:122] "RemoveContainer" containerID="193817d4f7d614f7af7c0000c733dff28e8c832ae9d791b804de621f31121956"
Nov 02 22:00:16.613191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.613182    4528 scope.go:122] "RemoveContainer" containerID="7b104d3ebb338a59986b164007aa0ffe2fd682cc5f45cf73a79f245e869ead12"
Nov 02 22:00:16.613554 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:16.613533    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng_odh-ai-gateway-infra(8375e961-9f74-44ce-9569-7d596dcd5e01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01"
Nov 02 22:00:16.615828 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.615797    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" event={"ID":"ab7eaa8e-01e8-481f-9523-49b9bcc2a987","Type":"ContainerStarted","Data":"0cce18a9e0eb897644543cb89f13e733e74a425137a288c7a29f69456c2cc0e6"}
Nov 02 22:00:16.617542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.617521    4528 generic.go:356] "Generic (PLEG): container finished" podID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerID="eeeb0853ec744bbc4f915a7c5c3fad8db016d1c66eff5e95b994efd33916a818" exitCode=60
Nov 02 22:00:16.617648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.617597    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" event={"ID":"91893bc4-e917-4f23-bc3f-a1ae147a6e1b","Type":"ContainerDied","Data":"eeeb0853ec744bbc4f915a7c5c3fad8db016d1c66eff5e95b994efd33916a818"}
Nov 02 22:00:16.617948 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.617927    4528 scope.go:122] "RemoveContainer" containerID="eeeb0853ec744bbc4f915a7c5c3fad8db016d1c66eff5e95b994efd33916a818"
Nov 02 22:00:16.618260 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:16.618237    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k_odh-ai-gateway-infra(91893bc4-e917-4f23-bc3f-a1ae147a6e1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b"
Nov 02 22:00:16.619645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.619625    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" event={"ID":"769fb8a4-79d9-49e4-bd13-47a4d114dd0c","Type":"ContainerStarted","Data":"20e03c1f468631ad2108a3f5c22bdc9b2d27f8218d847ebc0bfd6f8698592915"}
Nov 02 22:00:16.621587 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.621563    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" event={"ID":"8a52647f-17c7-47de-9773-2e54142b11a3","Type":"ContainerStarted","Data":"c4345c4198fae35f270a23ad13ad5a91f6682ea09f4cbb8269190890b24b830e"}
Nov 02 22:00:16.623366 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.623342    4528 generic.go:356] "Generic (PLEG): container finished" podID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerID="108d07cef3c68bffe4c196dd5cb855a7d044a72146b7b1277a0039d08895b208" exitCode=60
Nov 02 22:00:16.623454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.623409    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" event={"ID":"a3f38da4-e241-4598-92bd-95c9085e9cdc","Type":"ContainerDied","Data":"108d07cef3c68bffe4c196dd5cb855a7d044a72146b7b1277a0039d08895b208"}
Nov 02 22:00:16.623687 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.623671    4528 scope.go:122] "RemoveContainer" containerID="108d07cef3c68bffe4c196dd5cb855a7d044a72146b7b1277a0039d08895b208"
Nov 02 22:00:16.623934 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:16.623911    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb_odh-ai-gateway-infra(a3f38da4-e241-4598-92bd-95c9085e9cdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc"
Nov 02 22:00:16.625282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.625259    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" event={"ID":"3c3de28a-ef2f-4efc-8611-47e5baf0a857","Type":"ContainerStarted","Data":"c0feb3f264325ba11856923047bc1c2324d194ebb79c5c63143c64ae5064e833"}
Nov 02 22:00:16.627038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.627021    4528 scope.go:122] "RemoveContainer" containerID="19f1315d2775719040974043b420053b83e19ba898929a2fc029ffea49e85bc5"
Nov 02 22:00:16.639212 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.639198    4528 scope.go:122] "RemoveContainer" containerID="b5aadc338a499d8701e31f5ecf23433517d46e1c09b7e1c86afda13ad10b2560"
Nov 02 22:00:16.718485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.718415    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"]
Nov 02 22:00:16.831538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.831508    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"]
Nov 02 22:00:16.906575 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.906548    4528 scope.go:122] "RemoveContainer" containerID="6f278f1efa6a8a8d57a1d07e227b48cc07ee66e98802d65e91a5fa82f70e6dfd"
Nov 02 22:00:16.918330 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.918291    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ea82f2-85a7-4e4a-bd18-1ef77dacb533" path="/var/lib/kubelet/pods/08ea82f2-85a7-4e4a-bd18-1ef77dacb533/volumes"
Nov 02 22:00:16.918877 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:16.918838    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2f8896-050b-4a6b-a685-2975ac72b63f" path="/var/lib/kubelet/pods/2c2f8896-050b-4a6b-a685-2975ac72b63f/volumes"
Nov 02 22:00:17.057012 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.056985    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"]
Nov 02 22:00:17.634459 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.634427    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" event={"ID":"84e3dd64-957e-4d5f-a7b1-892e58818eb3","Type":"ContainerStarted","Data":"906b1ce19b7a18641a6d542d8f52a1665e434fca534cb89d6e518676f8353b28"}
Nov 02 22:00:17.638042 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.638011    4528 generic.go:356] "Generic (PLEG): container finished" podID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerID="0cce18a9e0eb897644543cb89f13e733e74a425137a288c7a29f69456c2cc0e6" exitCode=60
Nov 02 22:00:17.638151 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.638080    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" event={"ID":"ab7eaa8e-01e8-481f-9523-49b9bcc2a987","Type":"ContainerDied","Data":"0cce18a9e0eb897644543cb89f13e733e74a425137a288c7a29f69456c2cc0e6"}
Nov 02 22:00:17.638238 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.638157    4528 scope.go:122] "RemoveContainer" containerID="3ee14b016f7902ac65d89bdaaf02eb95430cba4bf7ffdfa90740c6f4ae986b95"
Nov 02 22:00:17.638544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.638519    4528 scope.go:122] "RemoveContainer" containerID="0cce18a9e0eb897644543cb89f13e733e74a425137a288c7a29f69456c2cc0e6"
Nov 02 22:00:17.638902 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:17.638848    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8_odh-ai-gateway-infra(ab7eaa8e-01e8-481f-9523-49b9bcc2a987)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987"
Nov 02 22:00:17.646893 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.646871    4528 generic.go:356] "Generic (PLEG): container finished" podID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerID="20e03c1f468631ad2108a3f5c22bdc9b2d27f8218d847ebc0bfd6f8698592915" exitCode=60
Nov 02 22:00:17.647013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.646960    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" event={"ID":"769fb8a4-79d9-49e4-bd13-47a4d114dd0c","Type":"ContainerDied","Data":"20e03c1f468631ad2108a3f5c22bdc9b2d27f8218d847ebc0bfd6f8698592915"}
Nov 02 22:00:17.647451 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.647428    4528 scope.go:122] "RemoveContainer" containerID="20e03c1f468631ad2108a3f5c22bdc9b2d27f8218d847ebc0bfd6f8698592915"
Nov 02 22:00:17.647732 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:17.647708    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr_odh-ai-gateway-infra(769fb8a4-79d9-49e4-bd13-47a4d114dd0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c"
Nov 02 22:00:17.649062 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.649037    4528 generic.go:356] "Generic (PLEG): container finished" podID="8a52647f-17c7-47de-9773-2e54142b11a3" containerID="c4345c4198fae35f270a23ad13ad5a91f6682ea09f4cbb8269190890b24b830e" exitCode=60
Nov 02 22:00:17.649192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.649105    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" event={"ID":"8a52647f-17c7-47de-9773-2e54142b11a3","Type":"ContainerDied","Data":"c4345c4198fae35f270a23ad13ad5a91f6682ea09f4cbb8269190890b24b830e"}
Nov 02 22:00:17.649457 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.649439    4528 scope.go:122] "RemoveContainer" containerID="c4345c4198fae35f270a23ad13ad5a91f6682ea09f4cbb8269190890b24b830e"
Nov 02 22:00:17.649715 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:17.649688    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz_odh-ai-gateway-infra(8a52647f-17c7-47de-9773-2e54142b11a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" podUID="8a52647f-17c7-47de-9773-2e54142b11a3"
Nov 02 22:00:17.654216 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.654196    4528 generic.go:356] "Generic (PLEG): container finished" podID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerID="c0feb3f264325ba11856923047bc1c2324d194ebb79c5c63143c64ae5064e833" exitCode=60
Nov 02 22:00:17.654322 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.654246    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" event={"ID":"3c3de28a-ef2f-4efc-8611-47e5baf0a857","Type":"ContainerDied","Data":"c0feb3f264325ba11856923047bc1c2324d194ebb79c5c63143c64ae5064e833"}
Nov 02 22:00:17.654579 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.654563    4528 scope.go:122] "RemoveContainer" containerID="c0feb3f264325ba11856923047bc1c2324d194ebb79c5c63143c64ae5064e833"
Nov 02 22:00:17.654795 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:17.654775    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9_odh-ai-gateway-infra(3c3de28a-ef2f-4efc-8611-47e5baf0a857)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857"
Nov 02 22:00:17.657691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.657674    4528 scope.go:122] "RemoveContainer" containerID="3b792e2b9161606be9df284b9a5024c8f81c3132bb3b769af503c104676076ed"
Nov 02 22:00:17.673295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.673273    4528 scope.go:122] "RemoveContainer" containerID="bd36e05bcb43bcc395893d456087069bf4dbe712c9d1f99867417bcf602d7d0c"
Nov 02 22:00:17.689784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.689700    4528 scope.go:122] "RemoveContainer" containerID="abfc7ea7941d77e8bccbc02f10b622a539d9cc5ed27778aff78563af6bd3aa6a"
Nov 02 22:00:17.853357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.853329    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"]
Nov 02 22:00:17.876609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.876584    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:17.880628 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.880608    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:17.887039 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.886991    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:17.906641 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.906619    4528 scope.go:122] "RemoveContainer" containerID="00f0e9be56132218fdbf9bbf02ed959e4a8b8b42db589a3f8be2ebdc7da99a17"
Nov 02 22:00:17.906755 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.906738    4528 scope.go:122] "RemoveContainer" containerID="ec7d63ca4500ce97127c7ad8930a20ea71d0276b0519449552d03c4d818a23b0"
Nov 02 22:00:17.940010 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.939990    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f38da4-e241-4598-92bd-95c9085e9cdc-kube-api-access-bg58k\" (UniqueName: \"kubernetes.io/projected/a3f38da4-e241-4598-92bd-95c9085e9cdc-kube-api-access-bg58k\") pod \"a3f38da4-e241-4598-92bd-95c9085e9cdc\" (UID: \"a3f38da4-e241-4598-92bd-95c9085e9cdc\") "
Nov 02 22:00:17.940161 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.940046    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-maas-api-service-ca\") pod \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\" (UID: \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\") "
Nov 02 22:00:17.940161 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.940089    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8375e961-9f74-44ce-9569-7d596dcd5e01-kube-api-access-ftgxv\" (UniqueName: \"kubernetes.io/projected/8375e961-9f74-44ce-9569-7d596dcd5e01-kube-api-access-ftgxv\") pod \"8375e961-9f74-44ce-9569-7d596dcd5e01\" (UID: \"8375e961-9f74-44ce-9569-7d596dcd5e01\") "
Nov 02 22:00:17.940304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.940281    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8375e961-9f74-44ce-9569-7d596dcd5e01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8375e961-9f74-44ce-9569-7d596dcd5e01-maas-api-service-ca\") pod \"8375e961-9f74-44ce-9569-7d596dcd5e01\" (UID: \"8375e961-9f74-44ce-9569-7d596dcd5e01\") "
Nov 02 22:00:17.940368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.940325    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f38da4-e241-4598-92bd-95c9085e9cdc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f38da4-e241-4598-92bd-95c9085e9cdc-maas-api-service-ca\") pod \"a3f38da4-e241-4598-92bd-95c9085e9cdc\" (UID: \"a3f38da4-e241-4598-92bd-95c9085e9cdc\") "
Nov 02 22:00:17.940429 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.940389    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-kube-api-access-qnxzm\" (UniqueName: \"kubernetes.io/projected/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-kube-api-access-qnxzm\") pod \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\" (UID: \"91893bc4-e917-4f23-bc3f-a1ae147a6e1b\") "
Nov 02 22:00:17.940484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.940419    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-maas-api-service-ca" pod "91893bc4-e917-4f23-bc3f-a1ae147a6e1b" (UID: "91893bc4-e917-4f23-bc3f-a1ae147a6e1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:17.940635 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.940611    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8375e961-9f74-44ce-9569-7d596dcd5e01-maas-api-service-ca" pod "8375e961-9f74-44ce-9569-7d596dcd5e01" (UID: "8375e961-9f74-44ce-9569-7d596dcd5e01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:17.940726 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.940704    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f38da4-e241-4598-92bd-95c9085e9cdc-maas-api-service-ca" pod "a3f38da4-e241-4598-92bd-95c9085e9cdc" (UID: "a3f38da4-e241-4598-92bd-95c9085e9cdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:17.941413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.941376    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:17.941413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.941403    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8375e961-9f74-44ce-9569-7d596dcd5e01-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:17.941553 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.941419    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f38da4-e241-4598-92bd-95c9085e9cdc-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:17.942712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.942691    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f38da4-e241-4598-92bd-95c9085e9cdc-kube-api-access-bg58k" pod "a3f38da4-e241-4598-92bd-95c9085e9cdc" (UID: "a3f38da4-e241-4598-92bd-95c9085e9cdc"). InnerVolumeSpecName "kube-api-access-bg58k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:17.942952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.942931    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-kube-api-access-qnxzm" pod "91893bc4-e917-4f23-bc3f-a1ae147a6e1b" (UID: "91893bc4-e917-4f23-bc3f-a1ae147a6e1b"). InnerVolumeSpecName "kube-api-access-qnxzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:17.943325 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:17.943308    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8375e961-9f74-44ce-9569-7d596dcd5e01-kube-api-access-ftgxv" pod "8375e961-9f74-44ce-9569-7d596dcd5e01" (UID: "8375e961-9f74-44ce-9569-7d596dcd5e01"). InnerVolumeSpecName "kube-api-access-ftgxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:18.042878 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.042850    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgxv\" (UniqueName: \"kubernetes.io/projected/8375e961-9f74-44ce-9569-7d596dcd5e01-kube-api-access-ftgxv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:18.042878 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.042879    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnxzm\" (UniqueName: \"kubernetes.io/projected/91893bc4-e917-4f23-bc3f-a1ae147a6e1b-kube-api-access-qnxzm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:18.043047 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.042891    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg58k\" (UniqueName: \"kubernetes.io/projected/a3f38da4-e241-4598-92bd-95c9085e9cdc-kube-api-access-bg58k\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:18.080455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.080426    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"]
Nov 02 22:00:18.210104 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.210014    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"]
Nov 02 22:00:18.419242 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.419210    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"]
Nov 02 22:00:18.666696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.666662    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb" event={"ID":"a3f38da4-e241-4598-92bd-95c9085e9cdc","Type":"ContainerDied","Data":"708c3a00bbe9372c743862424a24ed33f40da253b8bb05bfd62ac333b5362a4a"}
Nov 02 22:00:18.667141 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.666681    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"
Nov 02 22:00:18.667141 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.666764    4528 scope.go:122] "RemoveContainer" containerID="108d07cef3c68bffe4c196dd5cb855a7d044a72146b7b1277a0039d08895b208"
Nov 02 22:00:18.679244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.679211    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" event={"ID":"26ba9ae2-aa81-4d1d-90db-6d9890c513c3","Type":"ContainerStarted","Data":"9d879ea672721096f3eb11f532cbb9c6fea1a139551edf0f875d220030c1fbef"}
Nov 02 22:00:18.684939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.684893    4528 generic.go:356] "Generic (PLEG): container finished" podID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerID="906b1ce19b7a18641a6d542d8f52a1665e434fca534cb89d6e518676f8353b28" exitCode=60
Nov 02 22:00:18.685051 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.685023    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" event={"ID":"84e3dd64-957e-4d5f-a7b1-892e58818eb3","Type":"ContainerDied","Data":"906b1ce19b7a18641a6d542d8f52a1665e434fca534cb89d6e518676f8353b28"}
Nov 02 22:00:18.685477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.685453    4528 scope.go:122] "RemoveContainer" containerID="906b1ce19b7a18641a6d542d8f52a1665e434fca534cb89d6e518676f8353b28"
Nov 02 22:00:18.685808 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:18.685779    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894280-g58rv_odh-ai-gateway-infra(84e3dd64-957e-4d5f-a7b1-892e58818eb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3"
Nov 02 22:00:18.688722 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.688694    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng" event={"ID":"8375e961-9f74-44ce-9569-7d596dcd5e01","Type":"ContainerDied","Data":"f8ea6937e6b185f7ce9af8468864f14241ded7a2eaa0250af8016ecac6af6204"}
Nov 02 22:00:18.688861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.688844    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"
Nov 02 22:00:18.700452 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.699772    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k" event={"ID":"91893bc4-e917-4f23-bc3f-a1ae147a6e1b","Type":"ContainerDied","Data":"5fa15e70e610bde1c793490dfa217b5cf27a3a31dc64f5142cdcbd4170f8b1a3"}
Nov 02 22:00:18.700452 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.699879    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"
Nov 02 22:00:18.703394 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.703249    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" event={"ID":"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d","Type":"ContainerStarted","Data":"880b3c855bc6a0a5dd953a5583f59b73f9817c715fb8667f3b2f35b6e6f5b8bf"}
Nov 02 22:00:18.706072 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.705315    4528 scope.go:122] "RemoveContainer" containerID="6f278f1efa6a8a8d57a1d07e227b48cc07ee66e98802d65e91a5fa82f70e6dfd"
Nov 02 22:00:18.735038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.734595    4528 scope.go:122] "RemoveContainer" containerID="7b104d3ebb338a59986b164007aa0ffe2fd682cc5f45cf73a79f245e869ead12"
Nov 02 22:00:18.748700 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.748673    4528 scope.go:122] "RemoveContainer" containerID="eeeb0853ec744bbc4f915a7c5c3fad8db016d1c66eff5e95b994efd33916a818"
Nov 02 22:00:18.767176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:18.767150    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"]
Nov 02 22:00:19.038882 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.038854    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"]
Nov 02 22:00:19.045897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.045875    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894280-v2zng"]
Nov 02 22:00:19.082969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.082937    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:19.086157 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.086139    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:19.089961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.089944    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:19.093643 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.093624    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:19.164079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164048    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3de28a-ef2f-4efc-8611-47e5baf0a857-kube-api-access-zkcg9\" (UniqueName: \"kubernetes.io/projected/3c3de28a-ef2f-4efc-8611-47e5baf0a857-kube-api-access-zkcg9\") pod \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\" (UID: \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\") "
Nov 02 22:00:19.164204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164133    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a52647f-17c7-47de-9773-2e54142b11a3-kube-api-access-7g4c9\" (UniqueName: \"kubernetes.io/projected/8a52647f-17c7-47de-9773-2e54142b11a3-kube-api-access-7g4c9\") pod \"8a52647f-17c7-47de-9773-2e54142b11a3\" (UID: \"8a52647f-17c7-47de-9773-2e54142b11a3\") "
Nov 02 22:00:19.164204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164193    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a52647f-17c7-47de-9773-2e54142b11a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a52647f-17c7-47de-9773-2e54142b11a3-maas-api-service-ca\") pod \"8a52647f-17c7-47de-9773-2e54142b11a3\" (UID: \"8a52647f-17c7-47de-9773-2e54142b11a3\") "
Nov 02 22:00:19.164326 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164286    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-maas-api-service-ca\") pod \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\" (UID: \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\") "
Nov 02 22:00:19.164433 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164413    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3de28a-ef2f-4efc-8611-47e5baf0a857-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3de28a-ef2f-4efc-8611-47e5baf0a857-maas-api-service-ca\") pod \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\" (UID: \"3c3de28a-ef2f-4efc-8611-47e5baf0a857\") "
Nov 02 22:00:19.164497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164471    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-kube-api-access-hpzsz\" (UniqueName: \"kubernetes.io/projected/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-kube-api-access-hpzsz\") pod \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\" (UID: \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\") "
Nov 02 22:00:19.164551 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164535    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-maas-api-service-ca\") pod \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\" (UID: \"ab7eaa8e-01e8-481f-9523-49b9bcc2a987\") "
Nov 02 22:00:19.164652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164585    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a52647f-17c7-47de-9773-2e54142b11a3-maas-api-service-ca" pod "8a52647f-17c7-47de-9773-2e54142b11a3" (UID: "8a52647f-17c7-47de-9773-2e54142b11a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:19.164652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164599    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-kube-api-access-jmvdl\" (UniqueName: \"kubernetes.io/projected/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-kube-api-access-jmvdl\") pod \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\" (UID: \"769fb8a4-79d9-49e4-bd13-47a4d114dd0c\") "
Nov 02 22:00:19.164751 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164677    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-maas-api-service-ca" pod "769fb8a4-79d9-49e4-bd13-47a4d114dd0c" (UID: "769fb8a4-79d9-49e4-bd13-47a4d114dd0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:19.164945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.164925    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-maas-api-service-ca" pod "ab7eaa8e-01e8-481f-9523-49b9bcc2a987" (UID: "ab7eaa8e-01e8-481f-9523-49b9bcc2a987"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:19.165147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.165104    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3de28a-ef2f-4efc-8611-47e5baf0a857-maas-api-service-ca" pod "3c3de28a-ef2f-4efc-8611-47e5baf0a857" (UID: "3c3de28a-ef2f-4efc-8611-47e5baf0a857"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:19.165271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.165258    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a52647f-17c7-47de-9773-2e54142b11a3-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:19.165334 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.165279    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:19.165334 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.165293    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3de28a-ef2f-4efc-8611-47e5baf0a857-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:19.165334 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.165307    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:19.166390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.166368    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3de28a-ef2f-4efc-8611-47e5baf0a857-kube-api-access-zkcg9" pod "3c3de28a-ef2f-4efc-8611-47e5baf0a857" (UID: "3c3de28a-ef2f-4efc-8611-47e5baf0a857"). InnerVolumeSpecName "kube-api-access-zkcg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:19.166850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.166829    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-kube-api-access-jmvdl" pod "769fb8a4-79d9-49e4-bd13-47a4d114dd0c" (UID: "769fb8a4-79d9-49e4-bd13-47a4d114dd0c"). InnerVolumeSpecName "kube-api-access-jmvdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:19.166940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.166920    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a52647f-17c7-47de-9773-2e54142b11a3-kube-api-access-7g4c9" pod "8a52647f-17c7-47de-9773-2e54142b11a3" (UID: "8a52647f-17c7-47de-9773-2e54142b11a3"). InnerVolumeSpecName "kube-api-access-7g4c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:19.166977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.166950    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-kube-api-access-hpzsz" pod "ab7eaa8e-01e8-481f-9523-49b9bcc2a987" (UID: "ab7eaa8e-01e8-481f-9523-49b9bcc2a987"). InnerVolumeSpecName "kube-api-access-hpzsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:19.266285 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.266253    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkcg9\" (UniqueName: \"kubernetes.io/projected/3c3de28a-ef2f-4efc-8611-47e5baf0a857-kube-api-access-zkcg9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:19.266285 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.266288    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g4c9\" (UniqueName: \"kubernetes.io/projected/8a52647f-17c7-47de-9773-2e54142b11a3-kube-api-access-7g4c9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:19.266538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.266302    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpzsz\" (UniqueName: \"kubernetes.io/projected/ab7eaa8e-01e8-481f-9523-49b9bcc2a987-kube-api-access-hpzsz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:19.266538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.266315    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvdl\" (UniqueName: \"kubernetes.io/projected/769fb8a4-79d9-49e4-bd13-47a4d114dd0c-kube-api-access-jmvdl\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:19.379585 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.379552    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"]
Nov 02 22:00:19.382892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.382870    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894280-bqctb"]
Nov 02 22:00:19.720872 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.720777    4528 generic.go:356] "Generic (PLEG): container finished" podID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerID="880b3c855bc6a0a5dd953a5583f59b73f9817c715fb8667f3b2f35b6e6f5b8bf" exitCode=60
Nov 02 22:00:19.721351 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.720861    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" event={"ID":"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d","Type":"ContainerDied","Data":"880b3c855bc6a0a5dd953a5583f59b73f9817c715fb8667f3b2f35b6e6f5b8bf"}
Nov 02 22:00:19.721351 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.721032    4528 scope.go:122] "RemoveContainer" containerID="ec7d63ca4500ce97127c7ad8930a20ea71d0276b0519449552d03c4d818a23b0"
Nov 02 22:00:19.721427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.721378    4528 scope.go:122] "RemoveContainer" containerID="880b3c855bc6a0a5dd953a5583f59b73f9817c715fb8667f3b2f35b6e6f5b8bf"
Nov 02 22:00:19.721779 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:19.721741    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25_odh-ai-gateway-infra(fb0bfed4-b5c9-4f30-b136-87b8ad130d5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d"
Nov 02 22:00:19.723258 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.723236    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"
Nov 02 22:00:19.723359 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.723278    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr" event={"ID":"769fb8a4-79d9-49e4-bd13-47a4d114dd0c","Type":"ContainerDied","Data":"e028e868699baacb17b338d8ae72b2c94c481330c083462fb831d38d5658ad77"}
Nov 02 22:00:19.724801 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.724783    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"
Nov 02 22:00:19.724949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.724801    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz" event={"ID":"8a52647f-17c7-47de-9773-2e54142b11a3","Type":"ContainerDied","Data":"3d1ec34e9415ec45a9b9fe3a66a6307f43df4d0a765d0026851b3d010737db31"}
Nov 02 22:00:19.728717 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.728691    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9" event={"ID":"3c3de28a-ef2f-4efc-8611-47e5baf0a857","Type":"ContainerDied","Data":"63e974e7c42780e7400e3ee820ba1e5f64cf0faca4eb9c854626ef50dabc0c98"}
Nov 02 22:00:19.728818 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.728700    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"
Nov 02 22:00:19.734654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.734470    4528 generic.go:356] "Generic (PLEG): container finished" podID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerID="9d879ea672721096f3eb11f532cbb9c6fea1a139551edf0f875d220030c1fbef" exitCode=60
Nov 02 22:00:19.734654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.734540    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" event={"ID":"26ba9ae2-aa81-4d1d-90db-6d9890c513c3","Type":"ContainerDied","Data":"9d879ea672721096f3eb11f532cbb9c6fea1a139551edf0f875d220030c1fbef"}
Nov 02 22:00:19.734922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.734902    4528 scope.go:122] "RemoveContainer" containerID="9d879ea672721096f3eb11f532cbb9c6fea1a139551edf0f875d220030c1fbef"
Nov 02 22:00:19.735233 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:19.735207    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j_odh-ai-gateway-infra(26ba9ae2-aa81-4d1d-90db-6d9890c513c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3"
Nov 02 22:00:19.737922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.737904    4528 scope.go:122] "RemoveContainer" containerID="20e03c1f468631ad2108a3f5c22bdc9b2d27f8218d847ebc0bfd6f8698592915"
Nov 02 22:00:19.739167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.739077    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8" event={"ID":"ab7eaa8e-01e8-481f-9523-49b9bcc2a987","Type":"ContainerDied","Data":"170b2d96453ac28e67ce3a8215270c495743dfe4dca3039bd51f81f85be5b51b"}
Nov 02 22:00:19.739167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.739084    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"
Nov 02 22:00:19.749897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.749866    4528 scope.go:122] "RemoveContainer" containerID="c4345c4198fae35f270a23ad13ad5a91f6682ea09f4cbb8269190890b24b830e"
Nov 02 22:00:19.758846 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.758828    4528 scope.go:122] "RemoveContainer" containerID="c0feb3f264325ba11856923047bc1c2324d194ebb79c5c63143c64ae5064e833"
Nov 02 22:00:19.769232 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.769210    4528 scope.go:122] "RemoveContainer" containerID="00f0e9be56132218fdbf9bbf02ed959e4a8b8b42db589a3f8be2ebdc7da99a17"
Nov 02 22:00:19.785326 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.785307    4528 scope.go:122] "RemoveContainer" containerID="0cce18a9e0eb897644543cb89f13e733e74a425137a288c7a29f69456c2cc0e6"
Nov 02 22:00:19.831196 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.831167    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"]
Nov 02 22:00:19.834694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.834671    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894280-6n26k"]
Nov 02 22:00:19.879285 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.879265    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:19.974923 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.974860    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e3dd64-957e-4d5f-a7b1-892e58818eb3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e3dd64-957e-4d5f-a7b1-892e58818eb3-maas-api-service-ca\") pod \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\" (UID: \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\") "
Nov 02 22:00:19.975085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.975019    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e3dd64-957e-4d5f-a7b1-892e58818eb3-kube-api-access-sqlsq\" (UniqueName: \"kubernetes.io/projected/84e3dd64-957e-4d5f-a7b1-892e58818eb3-kube-api-access-sqlsq\") pod \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\" (UID: \"84e3dd64-957e-4d5f-a7b1-892e58818eb3\") "
Nov 02 22:00:19.975233 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.975209    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e3dd64-957e-4d5f-a7b1-892e58818eb3-maas-api-service-ca" pod "84e3dd64-957e-4d5f-a7b1-892e58818eb3" (UID: "84e3dd64-957e-4d5f-a7b1-892e58818eb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:19.975544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.975525    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e3dd64-957e-4d5f-a7b1-892e58818eb3-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:19.977021 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:19.977002    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e3dd64-957e-4d5f-a7b1-892e58818eb3-kube-api-access-sqlsq" pod "84e3dd64-957e-4d5f-a7b1-892e58818eb3" (UID: "84e3dd64-957e-4d5f-a7b1-892e58818eb3"). InnerVolumeSpecName "kube-api-access-sqlsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:20.077265 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.077234    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqlsq\" (UniqueName: \"kubernetes.io/projected/84e3dd64-957e-4d5f-a7b1-892e58818eb3-kube-api-access-sqlsq\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:20.220866 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.220832    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"]
Nov 02 22:00:20.224453 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.224426    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894280-q6sh8"]
Nov 02 22:00:20.505956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.505923    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"]
Nov 02 22:00:20.516150 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.516103    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894280-7cqj9"]
Nov 02 22:00:20.613543 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.613515    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"]
Nov 02 22:00:20.753213 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.753155    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv" event={"ID":"84e3dd64-957e-4d5f-a7b1-892e58818eb3","Type":"ContainerDied","Data":"79a92d31e8353b1fef349c54eaac21ebd95db8843f1718028449325520a663ae"}
Nov 02 22:00:20.753683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.753260    4528 scope.go:122] "RemoveContainer" containerID="906b1ce19b7a18641a6d542d8f52a1665e434fca534cb89d6e518676f8353b28"
Nov 02 22:00:20.753683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.753406    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"
Nov 02 22:00:20.803775 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.803734    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"]
Nov 02 22:00:20.806827 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.806799    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894280-4fcrr"]
Nov 02 22:00:20.902565 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.902544    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:20.916118 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.916089    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" path="/var/lib/kubelet/pods/3c3de28a-ef2f-4efc-8611-47e5baf0a857/volumes"
Nov 02 22:00:20.916569 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.916553    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" path="/var/lib/kubelet/pods/769fb8a4-79d9-49e4-bd13-47a4d114dd0c/volumes"
Nov 02 22:00:20.916927 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.916913    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" path="/var/lib/kubelet/pods/8375e961-9f74-44ce-9569-7d596dcd5e01/volumes"
Nov 02 22:00:20.917626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.917610    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" path="/var/lib/kubelet/pods/91893bc4-e917-4f23-bc3f-a1ae147a6e1b/volumes"
Nov 02 22:00:20.917973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.917961    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" path="/var/lib/kubelet/pods/a3f38da4-e241-4598-92bd-95c9085e9cdc/volumes"
Nov 02 22:00:20.918340 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.918328    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" path="/var/lib/kubelet/pods/ab7eaa8e-01e8-481f-9523-49b9bcc2a987/volumes"
Nov 02 22:00:20.952889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.952857    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"]
Nov 02 22:00:20.988312 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.988283    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-kube-api-access-x79tq\" (UniqueName: \"kubernetes.io/projected/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-kube-api-access-x79tq\") pod \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\" (UID: \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\") "
Nov 02 22:00:20.988488 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.988473    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-maas-api-service-ca\") pod \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\" (UID: \"26ba9ae2-aa81-4d1d-90db-6d9890c513c3\") "
Nov 02 22:00:20.988906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.988853    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-maas-api-service-ca" pod "26ba9ae2-aa81-4d1d-90db-6d9890c513c3" (UID: "26ba9ae2-aa81-4d1d-90db-6d9890c513c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:20.989667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.989586    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:20.990763 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:20.990738    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-kube-api-access-x79tq" pod "26ba9ae2-aa81-4d1d-90db-6d9890c513c3" (UID: "26ba9ae2-aa81-4d1d-90db-6d9890c513c3"). InnerVolumeSpecName "kube-api-access-x79tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:21.051358 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.051324    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"]
Nov 02 22:00:21.054454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.054428    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f619a351-bmnlx"]
Nov 02 22:00:21.088203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.088170    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"]
Nov 02 22:00:21.089150 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.089123    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894280-bxxbz"]
Nov 02 22:00:21.090851 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.090828    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x79tq\" (UniqueName: \"kubernetes.io/projected/26ba9ae2-aa81-4d1d-90db-6d9890c513c3-kube-api-access-x79tq\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:21.115905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.115884    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:21.192091 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.192062    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-kube-api-access-dhqm8\" (UniqueName: \"kubernetes.io/projected/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-kube-api-access-dhqm8\") pod \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\" (UID: \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\") "
Nov 02 22:00:21.192245 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.192152    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-maas-api-service-ca\") pod \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\" (UID: \"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d\") "
Nov 02 22:00:21.192547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.192524    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-maas-api-service-ca" pod "fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" (UID: "fb0bfed4-b5c9-4f30-b136-87b8ad130d5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:21.192673 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.192659    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:21.194191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.194168    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-kube-api-access-dhqm8" pod "fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" (UID: "fb0bfed4-b5c9-4f30-b136-87b8ad130d5d"). InnerVolumeSpecName "kube-api-access-dhqm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:21.294309 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.294285    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhqm8\" (UniqueName: \"kubernetes.io/projected/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d-kube-api-access-dhqm8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:21.307760 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.307730    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"]
Nov 02 22:00:21.311524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.311502    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-g58rv"]
Nov 02 22:00:21.778041 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.778016    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"
Nov 02 22:00:21.778516 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.778015    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j" event={"ID":"26ba9ae2-aa81-4d1d-90db-6d9890c513c3","Type":"ContainerDied","Data":"6af45bdb775eaffa124f8e7d0abc6b3ee88fca182b7ae1eb02fdf6728ad1589e"}
Nov 02 22:00:21.778516 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.778174    4528 scope.go:122] "RemoveContainer" containerID="9d879ea672721096f3eb11f532cbb9c6fea1a139551edf0f875d220030c1fbef"
Nov 02 22:00:21.780750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.780728    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25" event={"ID":"fb0bfed4-b5c9-4f30-b136-87b8ad130d5d","Type":"ContainerDied","Data":"51cba3052a53869a546248a33c258b9fec252827fdce6678d923f238cc0386ce"}
Nov 02 22:00:21.780856 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.780763    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"
Nov 02 22:00:21.789300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.789281    4528 scope.go:122] "RemoveContainer" containerID="880b3c855bc6a0a5dd953a5583f59b73f9817c715fb8667f3b2f35b6e6f5b8bf"
Nov 02 22:00:21.943181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.943151    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"]
Nov 02 22:00:21.947483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:21.947461    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894280-n268j"]
Nov 02 22:00:22.056327 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:22.056242    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"]
Nov 02 22:00:22.058910 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:22.058892    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894280-p2f25"]
Nov 02 22:00:22.921123 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:22.921081    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" path="/var/lib/kubelet/pods/26ba9ae2-aa81-4d1d-90db-6d9890c513c3/volumes"
Nov 02 22:00:22.921523 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:22.921509    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" path="/var/lib/kubelet/pods/84e3dd64-957e-4d5f-a7b1-892e58818eb3/volumes"
Nov 02 22:00:22.921873 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:22.921861    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" path="/var/lib/kubelet/pods/8a52647f-17c7-47de-9773-2e54142b11a3/volumes"
Nov 02 22:00:22.922513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:22.922497    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94bd94f6-eeb7-496c-9a76-68ce01b9275a" path="/var/lib/kubelet/pods/94bd94f6-eeb7-496c-9a76-68ce01b9275a/volumes"
Nov 02 22:00:22.922864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:22.922851    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" path="/var/lib/kubelet/pods/fb0bfed4-b5c9-4f30-b136-87b8ad130d5d/volumes"
Nov 02 22:00:27.841044 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:27.841010    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"]
Nov 02 22:00:27.841442 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:27.841261    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" containerID="cri-o://56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e" gracePeriod=30
Nov 02 22:00:30.735809 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:30.735776    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-chvg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:00:30.736185 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:30.735839    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:00:33.095675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.095646    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 22:00:33.119407 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119343    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-certs\") pod \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") "
Nov 02 22:00:33.119521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119430    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-podinfo\") pod \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") "
Nov 02 22:00:33.119521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119462    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-credential-socket\") pod \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") "
Nov 02 22:00:33.119521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119510    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-data\") pod \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") "
Nov 02 22:00:33.119681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119587    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-token\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-token\") pod \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") "
Nov 02 22:00:33.119681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119592    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-certs" pod "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" (UID: "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:33.119681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119657    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-kube-api-access-7tf4b\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-kube-api-access-7tf4b\") pod \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") "
Nov 02 22:00:33.119681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119676    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istiod-ca-cert\") pod \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") "
Nov 02 22:00:33.119897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119712    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-socket\") pod \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") "
Nov 02 22:00:33.119897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119754    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-envoy\") pod \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\" (UID: \"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7\") "
Nov 02 22:00:33.120066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119753    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-data" pod "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" (UID: "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:33.120066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119770    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-credential-socket" pod "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" (UID: "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:33.120066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.119964    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-socket" pod "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" (UID: "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:33.120066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.120022    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istiod-ca-cert" pod "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" (UID: "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:33.120451 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.120428    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:33.120494 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.120459    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:33.120494 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.120475    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:33.120494 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.120490    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:33.120591 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.120504    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:33.121675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.121648    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-token" pod "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" (UID: "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:33.121825 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.121808    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-envoy" pod "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" (UID: "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:33.122005 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.121982    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-podinfo" pod "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" (UID: "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:00:33.122119 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.121985    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-kube-api-access-7tf4b" pod "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" (UID: "683ffcec-8a51-4714-9a3e-a0f7c58a0ce7"). InnerVolumeSpecName "kube-api-access-7tf4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:33.222312 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.222283    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:33.222312 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.222309    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tf4b\" (UniqueName: \"kubernetes.io/projected/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-kube-api-access-7tf4b\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:33.222474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.222320    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:33.222474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.222329    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:33.879998 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.879966    4528 generic.go:356] "Generic (PLEG): container finished" podID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerID="56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e" exitCode=0
Nov 02 22:00:33.880298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.880013    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"
Nov 02 22:00:33.880298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.880054    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" event={"ID":"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7","Type":"ContainerDied","Data":"56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e"}
Nov 02 22:00:33.880298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.880143    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9" event={"ID":"683ffcec-8a51-4714-9a3e-a0f7c58a0ce7","Type":"ContainerDied","Data":"abdefbf2c1e481e9c4ff8bf8a0cead89ff708b6bcff5429db7c17633d6401b68"}
Nov 02 22:00:33.880298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.880176    4528 scope.go:122] "RemoveContainer" containerID="56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e"
Nov 02 22:00:33.890839 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.890823    4528 scope.go:122] "RemoveContainer" containerID="56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e"
Nov 02 22:00:33.891118 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:33.891075    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e\": container with ID starting with 56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e not found: ID does not exist" containerID="56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e"
Nov 02 22:00:33.891221 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:33.891123    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e"} err="failed to get container status \"56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e\": rpc error: code = NotFound desc = could not find container \"56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e\": container with ID starting with 56f42ff3361f1c773ee50d221470d918ac7f1c5310115edd62d07ff86d77e84e not found: ID does not exist"
Nov 02 22:00:34.049457 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:34.049426    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"]
Nov 02 22:00:34.052763 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:34.052736    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-chvg9"]
Nov 02 22:00:34.794199 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:34.794164    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"]
Nov 02 22:00:34.794627 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:34.794383    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy" containerID="cri-o://1f78bdb29269ad68601eaba931204cd4e95a1bbd6ce03676267197d41adc1571" gracePeriod=30
Nov 02 22:00:34.903897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:34.903862    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"]
Nov 02 22:00:34.904079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:34.904059    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy" containerID="cri-o://78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771" gracePeriod=30
Nov 02 22:00:34.920000 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:34.919961    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" path="/var/lib/kubelet/pods/683ffcec-8a51-4714-9a3e-a0f7c58a0ce7/volumes"
Nov 02 22:00:35.472858 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:35.472826    4528 scope.go:122] "RemoveContainer" containerID="7a203aa3d8b4c42e0bab77a8f42ffa40afd15340126597465f784fee622882eb"
Nov 02 22:00:39.508321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:39.508279    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:00:39.508703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:39.508338    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:00:39.558683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:39.558644    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:00:39.558821 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:39.558709    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:00:39.922845 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:39.922817    4528 generic.go:356] "Generic (PLEG): container finished" podID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerID="1f78bdb29269ad68601eaba931204cd4e95a1bbd6ce03676267197d41adc1571" exitCode=0
Nov 02 22:00:39.922965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:39.922891    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" event={"ID":"e8332a66-8360-47eb-83d5-a8059cc2875b","Type":"ContainerDied","Data":"1f78bdb29269ad68601eaba931204cd4e95a1bbd6ce03676267197d41adc1571"}
Nov 02 22:00:40.156742 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.156717    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 22:00:40.210484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210457    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-envoy\") pod \"646ad9a4-208c-4d2b-9577-7599a6426125\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") "
Nov 02 22:00:40.210484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210490    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-certs\") pod \"646ad9a4-208c-4d2b-9577-7599a6426125\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") "
Nov 02 22:00:40.210725 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210513    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-credential-socket\") pod \"646ad9a4-208c-4d2b-9577-7599a6426125\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") "
Nov 02 22:00:40.210725 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210543    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-kube-api-access-j8mtn\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-kube-api-access-j8mtn\") pod \"646ad9a4-208c-4d2b-9577-7599a6426125\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") "
Nov 02 22:00:40.210725 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210617    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646ad9a4-208c-4d2b-9577-7599a6426125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646ad9a4-208c-4d2b-9577-7599a6426125-istiod-ca-cert\") pod \"646ad9a4-208c-4d2b-9577-7599a6426125\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") "
Nov 02 22:00:40.210725 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210648    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-istio-token\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-istio-token\") pod \"646ad9a4-208c-4d2b-9577-7599a6426125\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") "
Nov 02 22:00:40.211023 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210751    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-data\") pod \"646ad9a4-208c-4d2b-9577-7599a6426125\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") "
Nov 02 22:00:40.211023 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210757    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-certs" pod "646ad9a4-208c-4d2b-9577-7599a6426125" (UID: "646ad9a4-208c-4d2b-9577-7599a6426125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.211023 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210806    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/646ad9a4-208c-4d2b-9577-7599a6426125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646ad9a4-208c-4d2b-9577-7599a6426125-istio-podinfo\") pod \"646ad9a4-208c-4d2b-9577-7599a6426125\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") "
Nov 02 22:00:40.211023 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210819    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-credential-socket" pod "646ad9a4-208c-4d2b-9577-7599a6426125" (UID: "646ad9a4-208c-4d2b-9577-7599a6426125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.211023 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.210944    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-socket\") pod \"646ad9a4-208c-4d2b-9577-7599a6426125\" (UID: \"646ad9a4-208c-4d2b-9577-7599a6426125\") "
Nov 02 22:00:40.211353 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.211023    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-data" pod "646ad9a4-208c-4d2b-9577-7599a6426125" (UID: "646ad9a4-208c-4d2b-9577-7599a6426125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.211353 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.211032    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646ad9a4-208c-4d2b-9577-7599a6426125-istiod-ca-cert" pod "646ad9a4-208c-4d2b-9577-7599a6426125" (UID: "646ad9a4-208c-4d2b-9577-7599a6426125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:40.211353 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.211303    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-socket" pod "646ad9a4-208c-4d2b-9577-7599a6426125" (UID: "646ad9a4-208c-4d2b-9577-7599a6426125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.211555 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.211534    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646ad9a4-208c-4d2b-9577-7599a6426125-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.211623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.211562    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.211623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.211576    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.211623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.211591    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.211623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.211605    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.212822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.212788    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-envoy" pod "646ad9a4-208c-4d2b-9577-7599a6426125" (UID: "646ad9a4-208c-4d2b-9577-7599a6426125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.212950 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.212880    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-istio-token" pod "646ad9a4-208c-4d2b-9577-7599a6426125" (UID: "646ad9a4-208c-4d2b-9577-7599a6426125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:40.213141 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.213093    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/646ad9a4-208c-4d2b-9577-7599a6426125-istio-podinfo" pod "646ad9a4-208c-4d2b-9577-7599a6426125" (UID: "646ad9a4-208c-4d2b-9577-7599a6426125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:00:40.213272 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.213250    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-kube-api-access-j8mtn" pod "646ad9a4-208c-4d2b-9577-7599a6426125" (UID: "646ad9a4-208c-4d2b-9577-7599a6426125"). InnerVolumeSpecName "kube-api-access-j8mtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:40.313670 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.313633    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.313670 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.313667    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646ad9a4-208c-4d2b-9577-7599a6426125-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.313885 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.313683    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646ad9a4-208c-4d2b-9577-7599a6426125-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.313885 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.313697    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8mtn\" (UniqueName: \"kubernetes.io/projected/646ad9a4-208c-4d2b-9577-7599a6426125-kube-api-access-j8mtn\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.446068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.446043    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 22:00:40.515870 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.515845    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8332a66-8360-47eb-83d5-a8059cc2875b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8332a66-8360-47eb-83d5-a8059cc2875b-istiod-ca-cert\") pod \"e8332a66-8360-47eb-83d5-a8059cc2875b\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") "
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.515876    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-kube-api-access-krdgg\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-kube-api-access-krdgg\") pod \"e8332a66-8360-47eb-83d5-a8059cc2875b\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") "
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.515931    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-token\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-token\") pod \"e8332a66-8360-47eb-83d5-a8059cc2875b\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") "
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.515970    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-credential-socket\") pod \"e8332a66-8360-47eb-83d5-a8059cc2875b\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") "
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516061    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-envoy\") pod \"e8332a66-8360-47eb-83d5-a8059cc2875b\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") "
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516087    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-socket\") pod \"e8332a66-8360-47eb-83d5-a8059cc2875b\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") "
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516143    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-certs\") pod \"e8332a66-8360-47eb-83d5-a8059cc2875b\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") "
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516273    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-data\") pod \"e8332a66-8360-47eb-83d5-a8059cc2875b\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") "
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516304    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-podinfo\") pod \"e8332a66-8360-47eb-83d5-a8059cc2875b\" (UID: \"e8332a66-8360-47eb-83d5-a8059cc2875b\") "
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516320    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-credential-socket" pod "e8332a66-8360-47eb-83d5-a8059cc2875b" (UID: "e8332a66-8360-47eb-83d5-a8059cc2875b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516412    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8332a66-8360-47eb-83d5-a8059cc2875b-istiod-ca-cert" pod "e8332a66-8360-47eb-83d5-a8059cc2875b" (UID: "e8332a66-8360-47eb-83d5-a8059cc2875b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516428    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-socket" pod "e8332a66-8360-47eb-83d5-a8059cc2875b" (UID: "e8332a66-8360-47eb-83d5-a8059cc2875b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516455    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-certs" pod "e8332a66-8360-47eb-83d5-a8059cc2875b" (UID: "e8332a66-8360-47eb-83d5-a8059cc2875b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.516547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516532    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-data" pod "e8332a66-8360-47eb-83d5-a8059cc2875b" (UID: "e8332a66-8360-47eb-83d5-a8059cc2875b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.517339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516848    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.517339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516870    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.517339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516885    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8332a66-8360-47eb-83d5-a8059cc2875b-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.517339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516906    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.517339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.516923    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.518046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.518021    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-kube-api-access-krdgg" pod "e8332a66-8360-47eb-83d5-a8059cc2875b" (UID: "e8332a66-8360-47eb-83d5-a8059cc2875b"). InnerVolumeSpecName "kube-api-access-krdgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:40.518179 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.518148    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-envoy" pod "e8332a66-8360-47eb-83d5-a8059cc2875b" (UID: "e8332a66-8360-47eb-83d5-a8059cc2875b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:00:40.518179 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.518141    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-token" pod "e8332a66-8360-47eb-83d5-a8059cc2875b" (UID: "e8332a66-8360-47eb-83d5-a8059cc2875b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:00:40.518455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.518439    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-podinfo" pod "e8332a66-8360-47eb-83d5-a8059cc2875b" (UID: "e8332a66-8360-47eb-83d5-a8059cc2875b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:00:40.618494 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.618418    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.618494 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.618445    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.618494 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.618453    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8332a66-8360-47eb-83d5-a8059cc2875b-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.618494 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.618462    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krdgg\" (UniqueName: \"kubernetes.io/projected/e8332a66-8360-47eb-83d5-a8059cc2875b-kube-api-access-krdgg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:00:40.929559 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.929533    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"
Nov 02 22:00:40.929692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.929563    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd" event={"ID":"e8332a66-8360-47eb-83d5-a8059cc2875b","Type":"ContainerDied","Data":"69b214116b50b21e8f0cd1d5b69095dfebba85a8d68833c3634087461796ef44"}
Nov 02 22:00:40.929692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.929614    4528 scope.go:122] "RemoveContainer" containerID="1f78bdb29269ad68601eaba931204cd4e95a1bbd6ce03676267197d41adc1571"
Nov 02 22:00:40.931079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.931052    4528 generic.go:356] "Generic (PLEG): container finished" podID="646ad9a4-208c-4d2b-9577-7599a6426125" containerID="78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771" exitCode=0
Nov 02 22:00:40.931230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.931096    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"
Nov 02 22:00:40.931230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.931149    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" event={"ID":"646ad9a4-208c-4d2b-9577-7599a6426125","Type":"ContainerDied","Data":"78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771"}
Nov 02 22:00:40.931230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.931190    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n" event={"ID":"646ad9a4-208c-4d2b-9577-7599a6426125","Type":"ContainerDied","Data":"c9082513d69d52461c75b8619c6b94a37ba6e0cafda21a98dbaaa98e9735df1d"}
Nov 02 22:00:40.939244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.939216    4528 scope.go:122] "RemoveContainer" containerID="78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771"
Nov 02 22:00:40.947450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.947434    4528 scope.go:122] "RemoveContainer" containerID="78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771"
Nov 02 22:00:40.947707 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:40.947690    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771\": container with ID starting with 78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771 not found: ID does not exist" containerID="78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771"
Nov 02 22:00:40.947766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:40.947718    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771"} err="failed to get container status \"78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771\": rpc error: code = NotFound desc = could not find container \"78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771\": container with ID starting with 78b4649bf747cdd86b2e8f6c6bc84acd6b1953d2732508e38973331f89369771 not found: ID does not exist"
Nov 02 22:00:41.095992 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:41.095956    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"]
Nov 02 22:00:41.146197 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:41.146168    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2dm6n"]
Nov 02 22:00:41.209709 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:41.209630    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"]
Nov 02 22:00:41.213781 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:41.213757    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-xffwd"]
Nov 02 22:00:42.916202 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:42.916165    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" path="/var/lib/kubelet/pods/646ad9a4-208c-4d2b-9577-7599a6426125/volumes"
Nov 02 22:00:42.916854 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:00:42.916832    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" path="/var/lib/kubelet/pods/e8332a66-8360-47eb-83d5-a8059cc2875b/volumes"
Nov 02 22:00:44.112554 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:00:44.112514    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:01:05.009968 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:05.009928    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"]
Nov 02 22:01:05.010585 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:05.010202    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy" containerID="cri-o://e27ab832b6fd64ac6e183b3097bf720364ab09042a3f9d954c6797a231c24b9a" gracePeriod=30
Nov 02 22:01:10.139306 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.139273    4528 generic.go:356] "Generic (PLEG): container finished" podID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerID="e27ab832b6fd64ac6e183b3097bf720364ab09042a3f9d954c6797a231c24b9a" exitCode=0
Nov 02 22:01:10.139654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.139345    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" event={"ID":"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0","Type":"ContainerDied","Data":"e27ab832b6fd64ac6e183b3097bf720364ab09042a3f9d954c6797a231c24b9a"}
Nov 02 22:01:10.673830 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.673805    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 22:01:10.774164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774134    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-socket\") pod \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") "
Nov 02 22:01:10.774308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774172    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-certs\") pod \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") "
Nov 02 22:01:10.774308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774261    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-credential-socket\") pod \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") "
Nov 02 22:01:10.774405 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774359    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-token\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-token\") pod \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") "
Nov 02 22:01:10.774405 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774391    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-podinfo\") pod \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") "
Nov 02 22:01:10.774521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774415    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-socket" pod "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" (UID: "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:01:10.774521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774446    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-envoy\") pod \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") "
Nov 02 22:01:10.774521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774480    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-kube-api-access-xvqtf\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-kube-api-access-xvqtf\") pod \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") "
Nov 02 22:01:10.774521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774506    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-certs" pod "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" (UID: "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:01:10.774740 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774513    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istiod-ca-cert\") pod \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") "
Nov 02 22:01:10.774740 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774580    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-credential-socket" pod "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" (UID: "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:01:10.774740 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774592    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-data\") pod \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\" (UID: \"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0\") "
Nov 02 22:01:10.774908 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774827    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istiod-ca-cert" pod "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" (UID: "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:01:10.774965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.774941    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-data" pod "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" (UID: "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:01:10.775312 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.775287    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:01:10.775431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.775316    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:01:10.775431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.775332    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:01:10.775431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.775346    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:01:10.775431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.775361    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:01:10.776668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.776572    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-token" pod "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" (UID: "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:01:10.776668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.776638    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-kube-api-access-xvqtf" pod "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" (UID: "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0"). InnerVolumeSpecName "kube-api-access-xvqtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:01:10.776668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.776648    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-envoy" pod "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" (UID: "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:01:10.776668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.776654    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-podinfo" pod "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" (UID: "ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:01:10.876031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.876001    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:01:10.876031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.876023    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:01:10.876031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.876034    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:01:10.876270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:10.876043    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvqtf\" (UniqueName: \"kubernetes.io/projected/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0-kube-api-access-xvqtf\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:01:11.149142 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:11.149054    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"
Nov 02 22:01:11.149142 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:11.149085    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6" event={"ID":"ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0","Type":"ContainerDied","Data":"38ad08de975a12fd47ed5f45a54fcd050a47a812d1120be5728d7385b2be1a0f"}
Nov 02 22:01:11.149668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:11.149160    4528 scope.go:122] "RemoveContainer" containerID="e27ab832b6fd64ac6e183b3097bf720364ab09042a3f9d954c6797a231c24b9a"
Nov 02 22:01:11.313842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:11.313809    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"]
Nov 02 22:01:11.317563 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:11.317539    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-bvlx6"]
Nov 02 22:01:12.914306 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:12.914275    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" path="/var/lib/kubelet/pods/ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0/volumes"
Nov 02 22:01:14.113053 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:01:14.113011    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:01:19.485844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:19.485807    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:01:35.552690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:01:35.552653    4528 scope.go:122] "RemoveContainer" containerID="e92efdd5b620dfc967df6dda55fd89735df6d8649c3965ab02d9fbd662fe2d45"
Nov 02 22:01:44.113034 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:01:44.112988    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:02:02.129919 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:02.129882    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:02:14.114487 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:02:14.114445    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:02:42.236499 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.236466    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"]
Nov 02 22:02:42.237072 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237058    4528 cpu_manager.go:414] "containerMap: removing container" podUID="08ea82f2-85a7-4e4a-bd18-1ef77dacb533" containerName="payload-pre-processing"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237073    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ea82f2-85a7-4e4a-bd18-1ef77dacb533" containerName="payload-pre-processing"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237091    4528 cpu_manager.go:414] "containerMap: removing container" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237096    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237104    4528 cpu_manager.go:414] "containerMap: removing container" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerName="cleanup"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237122    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerName="cleanup"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237129    4528 cpu_manager.go:414] "containerMap: removing container" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerName="cleanup"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237134    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerName="cleanup"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237144    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2c2f8896-050b-4a6b-a685-2975ac72b63f" containerName="payload-processing"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237148    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2f8896-050b-4a6b-a685-2975ac72b63f" containerName="payload-processing"
Nov 02 22:02:42.237152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237156    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237161    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237169    4528 cpu_manager.go:414] "containerMap: removing container" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237175    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237181    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237185    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237191    4528 cpu_manager.go:414] "containerMap: removing container" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237195    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237203    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237207    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237214    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237218    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237226    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237231    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237238    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237242    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237250    4528 cpu_manager.go:414] "containerMap: removing container" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237254    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237259    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237264    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237271    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237275    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237284    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237288    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237294    4528 cpu_manager.go:414] "containerMap: removing container" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237299    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237304    4528 cpu_manager.go:414] "containerMap: removing container" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237308    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237313    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237317    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237326    4528 cpu_manager.go:414] "containerMap: removing container" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237329    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237343    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237347    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237353    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237357    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237365    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237369    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237379    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237384    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237391    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237395    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237402    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237406    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237412    4528 cpu_manager.go:414] "containerMap: removing container" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237416    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237423    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" containerName="cleanup"
Nov 02 22:02:42.237485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237427    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237508    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237516    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237522    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237527    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237531    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237536    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237540    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237545    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237551    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237556    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237561    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237567    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237572    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8332a66-8360-47eb-83d5-a8059cc2875b" containerName="istio-proxy"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237577    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffdb0c3b-9b7b-43d0-aed7-fc8d516a4cb0" containerName="istio-proxy"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237582    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237586    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237591    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237596    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237600    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="683ffcec-8a51-4714-9a3e-a0f7c58a0ce7" containerName="istio-proxy"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237606    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237611    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237615    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237621    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ea82f2-85a7-4e4a-bd18-1ef77dacb533" containerName="payload-pre-processing"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237628    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="646ad9a4-208c-4d2b-9577-7599a6426125" containerName="istio-proxy"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237633    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237639    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237644    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2f8896-050b-4a6b-a685-2975ac72b63f" containerName="payload-processing"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237649    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerName="cleanup"
Nov 02 22:02:42.238934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.237655    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerName="cleanup"
Nov 02 22:02:42.241071 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.241055    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.294182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.294139    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"]
Nov 02 22:02:42.339741 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.339707    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:02:42.339984 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.339777    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:02:42.339984 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.339841    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:02:42.339984 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.339885    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:02:42.369372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.369340    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a898319-7458-4be4-898e-164e57dff464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64773fe5-nv7c6\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.369526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.369493    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a898319-7458-4be4-898e-164e57dff464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64773fe5-nv7c6\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.369608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.369593    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlhqd\" (UniqueName: \"kubernetes.io/projected/3a898319-7458-4be4-898e-164e57dff464-kube-api-access-jlhqd\") pod \"maas-api-revoke-sub-64773fe5-nv7c6\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.471043 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.471005    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a898319-7458-4be4-898e-164e57dff464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64773fe5-nv7c6\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.471273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.471136    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlhqd\" (UniqueName: \"kubernetes.io/projected/3a898319-7458-4be4-898e-164e57dff464-kube-api-access-jlhqd\") pod \"maas-api-revoke-sub-64773fe5-nv7c6\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.471273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.471191    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a898319-7458-4be4-898e-164e57dff464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64773fe5-nv7c6\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.471870 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.471840    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a898319-7458-4be4-898e-164e57dff464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64773fe5-nv7c6\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.473316 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.473291    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a898319-7458-4be4-898e-164e57dff464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64773fe5-nv7c6\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.528435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.528405    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlhqd\" (UniqueName: \"kubernetes.io/projected/3a898319-7458-4be4-898e-164e57dff464-kube-api-access-jlhqd\") pod \"maas-api-revoke-sub-64773fe5-nv7c6\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.555282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.555255    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:42.928024 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:02:42.927990    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a898319_7458_4be4_898e_164e57dff464.slice/crio-ad571d9241fffd8bd3a9c4cd7945e0cc2871a77b6aa0f7ef42c87c46fc4359e1 WatchSource:0}: Error finding container ad571d9241fffd8bd3a9c4cd7945e0cc2871a77b6aa0f7ef42c87c46fc4359e1: Status 404 returned error can't find the container with id ad571d9241fffd8bd3a9c4cd7945e0cc2871a77b6aa0f7ef42c87c46fc4359e1
Nov 02 22:02:42.929394 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:42.929368    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"]
Nov 02 22:02:43.769064 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:43.769030    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6" event={"ID":"3a898319-7458-4be4-898e-164e57dff464","Type":"ContainerStarted","Data":"214df7fa2338902e44084cdf21006e94da5d17033541db3b500cae87f7bf1dfd"}
Nov 02 22:02:43.769503 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:43.769085    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6" event={"ID":"3a898319-7458-4be4-898e-164e57dff464","Type":"ContainerStarted","Data":"ad571d9241fffd8bd3a9c4cd7945e0cc2871a77b6aa0f7ef42c87c46fc4359e1"}
Nov 02 22:02:43.889744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:43.889691    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6" podStartSLOduration=1.889677856 podStartE2EDuration="1.889677856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:43.888512892 +0000 UTC m=+3251.152049209" watchObservedRunningTime="2026-11-02 22:02:43.889677856 +0000 UTC m=+3251.153214173"
Nov 02 22:02:44.114781 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:02:44.114691    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:02:44.776279 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:44.776240    4528 generic.go:356] "Generic (PLEG): container finished" podID="3a898319-7458-4be4-898e-164e57dff464" containerID="214df7fa2338902e44084cdf21006e94da5d17033541db3b500cae87f7bf1dfd" exitCode=0
Nov 02 22:02:44.776678 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:44.776313    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6" event={"ID":"3a898319-7458-4be4-898e-164e57dff464","Type":"ContainerDied","Data":"214df7fa2338902e44084cdf21006e94da5d17033541db3b500cae87f7bf1dfd"}
Nov 02 22:02:46.043789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.043734    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:46.122946 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.122920    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a898319-7458-4be4-898e-164e57dff464-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a898319-7458-4be4-898e-164e57dff464-maas-api-openshift-ca\") pod \"3a898319-7458-4be4-898e-164e57dff464\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") "
Nov 02 22:02:46.123100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.123012    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a898319-7458-4be4-898e-164e57dff464-kube-api-access-jlhqd\" (UniqueName: \"kubernetes.io/projected/3a898319-7458-4be4-898e-164e57dff464-kube-api-access-jlhqd\") pod \"3a898319-7458-4be4-898e-164e57dff464\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") "
Nov 02 22:02:46.123210 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.123165    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a898319-7458-4be4-898e-164e57dff464-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a898319-7458-4be4-898e-164e57dff464-maas-api-xks-ca\") pod \"3a898319-7458-4be4-898e-164e57dff464\" (UID: \"3a898319-7458-4be4-898e-164e57dff464\") "
Nov 02 22:02:46.123320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.123292    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a898319-7458-4be4-898e-164e57dff464-maas-api-openshift-ca" pod "3a898319-7458-4be4-898e-164e57dff464" (UID: "3a898319-7458-4be4-898e-164e57dff464"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:02:46.123650 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.123631    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a898319-7458-4be4-898e-164e57dff464-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:02:46.125051 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.125033    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a898319-7458-4be4-898e-164e57dff464-maas-api-xks-ca" pod "3a898319-7458-4be4-898e-164e57dff464" (UID: "3a898319-7458-4be4-898e-164e57dff464"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:02:46.125216 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.125202    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a898319-7458-4be4-898e-164e57dff464-kube-api-access-jlhqd" pod "3a898319-7458-4be4-898e-164e57dff464" (UID: "3a898319-7458-4be4-898e-164e57dff464"). InnerVolumeSpecName "kube-api-access-jlhqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:02:46.224932 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.224886    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlhqd\" (UniqueName: \"kubernetes.io/projected/3a898319-7458-4be4-898e-164e57dff464-kube-api-access-jlhqd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:02:46.224932 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.224929    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a898319-7458-4be4-898e-164e57dff464-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:02:46.789958 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.789920    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"
Nov 02 22:02:46.790164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.789960    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6" event={"ID":"3a898319-7458-4be4-898e-164e57dff464","Type":"ContainerDied","Data":"ad571d9241fffd8bd3a9c4cd7945e0cc2871a77b6aa0f7ef42c87c46fc4359e1"}
Nov 02 22:02:46.790164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:46.790037    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad571d9241fffd8bd3a9c4cd7945e0cc2871a77b6aa0f7ef42c87c46fc4359e1"
Nov 02 22:02:52.044272 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.044243    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:02:52.976692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.976659    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"]
Nov 02 22:02:52.977239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977225    4528 cpu_manager.go:414] "containerMap: removing container" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerName="cleanup"
Nov 02 22:02:52.977298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977240    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e3dd64-957e-4d5f-a7b1-892e58818eb3" containerName="cleanup"
Nov 02 22:02:52.977298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977262    4528 cpu_manager.go:414] "containerMap: removing container" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerName="cleanup"
Nov 02 22:02:52.977298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977267    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerName="cleanup"
Nov 02 22:02:52.977298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977274    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerName="cleanup"
Nov 02 22:02:52.977298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977279    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerName="cleanup"
Nov 02 22:02:52.977298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977288    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerName="cleanup"
Nov 02 22:02:52.977298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977293    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977307    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3a898319-7458-4be4-898e-164e57dff464" containerName="revoke-keys"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977312    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a898319-7458-4be4-898e-164e57dff464" containerName="revoke-keys"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977329    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977333    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977344    4528 cpu_manager.go:414] "containerMap: removing container" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977349    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ba9ae2-aa81-4d1d-90db-6d9890c513c3" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977356    4528 cpu_manager.go:414] "containerMap: removing container" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977360    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769fb8a4-79d9-49e4-bd13-47a4d114dd0c" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977439    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a898319-7458-4be4-898e-164e57dff464" containerName="revoke-keys"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977447    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0bfed4-b5c9-4f30-b136-87b8ad130d5d" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977455    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3de28a-ef2f-4efc-8611-47e5baf0a857" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977460    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a52647f-17c7-47de-9773-2e54142b11a3" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977471    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f38da4-e241-4598-92bd-95c9085e9cdc" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977476    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8375e961-9f74-44ce-9569-7d596dcd5e01" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977481    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="91893bc4-e917-4f23-bc3f-a1ae147a6e1b" containerName="cleanup"
Nov 02 22:02:52.977568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.977486    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7eaa8e-01e8-481f-9523-49b9bcc2a987" containerName="cleanup"
Nov 02 22:02:52.981020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:52.981003    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.033570 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.033543    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:02:53.033857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.033832    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"]
Nov 02 22:02:53.079519 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.079488    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:02:53.079892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.079539    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:02:53.107425 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.107388    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b8aecf-j8cjw\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.107561 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.107441    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skt4x\" (UniqueName: \"kubernetes.io/projected/bcb8d3a0-7eac-49d7-829f-50e5133987f1-kube-api-access-skt4x\") pod \"maas-api-revoke-sub-96b8aecf-j8cjw\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.107671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.107650    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b8aecf-j8cjw\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.133905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.133868    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:02:53.209492 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.209457    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b8aecf-j8cjw\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.209730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.209509    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt4x\" (UniqueName: \"kubernetes.io/projected/bcb8d3a0-7eac-49d7-829f-50e5133987f1-kube-api-access-skt4x\") pod \"maas-api-revoke-sub-96b8aecf-j8cjw\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.209730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.209620    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b8aecf-j8cjw\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.210267 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.210239    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b8aecf-j8cjw\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.212093 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.212066    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b8aecf-j8cjw\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.266909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.266868    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skt4x\" (UniqueName: \"kubernetes.io/projected/bcb8d3a0-7eac-49d7-829f-50e5133987f1-kube-api-access-skt4x\") pod \"maas-api-revoke-sub-96b8aecf-j8cjw\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.294312 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.294275    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:53.665406 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:02:53.665379    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb8d3a0_7eac_49d7_829f_50e5133987f1.slice/crio-42617af3a1da90c85c5d3b18c35866be71cc16217fcedaec66d3752d6d70c4aa WatchSource:0}: Error finding container 42617af3a1da90c85c5d3b18c35866be71cc16217fcedaec66d3752d6d70c4aa: Status 404 returned error can't find the container with id 42617af3a1da90c85c5d3b18c35866be71cc16217fcedaec66d3752d6d70c4aa
Nov 02 22:02:53.667732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.667697    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"]
Nov 02 22:02:53.834545 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:53.834514    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw" event={"ID":"bcb8d3a0-7eac-49d7-829f-50e5133987f1","Type":"ContainerStarted","Data":"42617af3a1da90c85c5d3b18c35866be71cc16217fcedaec66d3752d6d70c4aa"}
Nov 02 22:02:54.841919 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:54.841823    4528 generic.go:356] "Generic (PLEG): container finished" podID="bcb8d3a0-7eac-49d7-829f-50e5133987f1" containerID="947b7f57d68e3a8723238cb83bd2d2567b649a4afa13f417fa1d15ee7227376f" exitCode=0
Nov 02 22:02:54.842314 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:54.841913    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw" event={"ID":"bcb8d3a0-7eac-49d7-829f-50e5133987f1","Type":"ContainerDied","Data":"947b7f57d68e3a8723238cb83bd2d2567b649a4afa13f417fa1d15ee7227376f"}
Nov 02 22:02:56.017230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.017200    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:56.147371 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.147273    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-xks-ca\") pod \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") "
Nov 02 22:02:56.147371 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.147353    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-openshift-ca\") pod \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") "
Nov 02 22:02:56.147660 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.147600    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb8d3a0-7eac-49d7-829f-50e5133987f1-kube-api-access-skt4x\" (UniqueName: \"kubernetes.io/projected/bcb8d3a0-7eac-49d7-829f-50e5133987f1-kube-api-access-skt4x\") pod \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\" (UID: \"bcb8d3a0-7eac-49d7-829f-50e5133987f1\") "
Nov 02 22:02:56.147822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.147791    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-openshift-ca" pod "bcb8d3a0-7eac-49d7-829f-50e5133987f1" (UID: "bcb8d3a0-7eac-49d7-829f-50e5133987f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:02:56.148451 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.148423    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:02:56.150745 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.150720    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-xks-ca" pod "bcb8d3a0-7eac-49d7-829f-50e5133987f1" (UID: "bcb8d3a0-7eac-49d7-829f-50e5133987f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:02:56.150880 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.150858    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb8d3a0-7eac-49d7-829f-50e5133987f1-kube-api-access-skt4x" pod "bcb8d3a0-7eac-49d7-829f-50e5133987f1" (UID: "bcb8d3a0-7eac-49d7-829f-50e5133987f1"). InnerVolumeSpecName "kube-api-access-skt4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:02:56.249862 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.249819    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb8d3a0-7eac-49d7-829f-50e5133987f1-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:02:56.249862 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.249855    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skt4x\" (UniqueName: \"kubernetes.io/projected/bcb8d3a0-7eac-49d7-829f-50e5133987f1-kube-api-access-skt4x\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:02:56.855601 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.855572    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"
Nov 02 22:02:56.855805 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.855602    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw" event={"ID":"bcb8d3a0-7eac-49d7-829f-50e5133987f1","Type":"ContainerDied","Data":"42617af3a1da90c85c5d3b18c35866be71cc16217fcedaec66d3752d6d70c4aa"}
Nov 02 22:02:56.855805 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:02:56.855677    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42617af3a1da90c85c5d3b18c35866be71cc16217fcedaec66d3752d6d70c4aa"
Nov 02 22:03:09.507830 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:09.507790    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"]
Nov 02 22:03:14.113801 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:03:14.113765    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:03:24.528101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.528059    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz" podUID="197d7371-cd9c-4fc4-959e-d319d38cf7dc" containerName="main" containerID="cri-o://64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c" gracePeriod=45
Nov 02 22:03:24.799442 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.799417    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 22:03:24.900312 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.900271    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tmp-dir\") pod \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") "
Nov 02 22:03:24.900521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.900427    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-model-cache\") pod \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") "
Nov 02 22:03:24.900521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.900490    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-dshm\") pod \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") "
Nov 02 22:03:24.900645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.900528    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-home\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-home\") pod \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") "
Nov 02 22:03:24.900645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.900546    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tls-certs\") pod \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") "
Nov 02 22:03:24.900645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.900546    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tmp-dir" pod "197d7371-cd9c-4fc4-959e-d319d38cf7dc" (UID: "197d7371-cd9c-4fc4-959e-d319d38cf7dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:24.900645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.900601    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197d7371-cd9c-4fc4-959e-d319d38cf7dc-kube-api-access-z8xxd\" (UniqueName: \"kubernetes.io/projected/197d7371-cd9c-4fc4-959e-d319d38cf7dc-kube-api-access-z8xxd\") pod \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\" (UID: \"197d7371-cd9c-4fc4-959e-d319d38cf7dc\") "
Nov 02 22:03:24.900884 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.900674    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-model-cache" pod "197d7371-cd9c-4fc4-959e-d319d38cf7dc" (UID: "197d7371-cd9c-4fc4-959e-d319d38cf7dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:24.900884 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.900768    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-home" pod "197d7371-cd9c-4fc4-959e-d319d38cf7dc" (UID: "197d7371-cd9c-4fc4-959e-d319d38cf7dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:24.901063 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.901040    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:24.901063 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.901061    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:24.901291 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.901071    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:24.902826 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.902801    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-dshm" pod "197d7371-cd9c-4fc4-959e-d319d38cf7dc" (UID: "197d7371-cd9c-4fc4-959e-d319d38cf7dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:24.902931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.902861    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tls-certs" pod "197d7371-cd9c-4fc4-959e-d319d38cf7dc" (UID: "197d7371-cd9c-4fc4-959e-d319d38cf7dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:03:24.902931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:24.902863    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197d7371-cd9c-4fc4-959e-d319d38cf7dc-kube-api-access-z8xxd" pod "197d7371-cd9c-4fc4-959e-d319d38cf7dc" (UID: "197d7371-cd9c-4fc4-959e-d319d38cf7dc"). InnerVolumeSpecName "kube-api-access-z8xxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:03:25.002756 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.002723    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d7371-cd9c-4fc4-959e-d319d38cf7dc-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:25.002988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.002972    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8xxd\" (UniqueName: \"kubernetes.io/projected/197d7371-cd9c-4fc4-959e-d319d38cf7dc-kube-api-access-z8xxd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:25.003105 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.003093    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d7371-cd9c-4fc4-959e-d319d38cf7dc-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:25.020795 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.020759    4528 generic.go:356] "Generic (PLEG): container finished" podID="197d7371-cd9c-4fc4-959e-d319d38cf7dc" containerID="64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c" exitCode=0
Nov 02 22:03:25.020951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.020821    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"
Nov 02 22:03:25.020951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.020840    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz" event={"ID":"197d7371-cd9c-4fc4-959e-d319d38cf7dc","Type":"ContainerDied","Data":"64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c"}
Nov 02 22:03:25.020951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.020894    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz" event={"ID":"197d7371-cd9c-4fc4-959e-d319d38cf7dc","Type":"ContainerDied","Data":"66deb0a4f6ccc386ce772778bdc00720b8c2c04fae1662f56e3ddd36d3bc631d"}
Nov 02 22:03:25.020951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.020911    4528 scope.go:122] "RemoveContainer" containerID="64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c"
Nov 02 22:03:25.030406 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.030377    4528 scope.go:122] "RemoveContainer" containerID="64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c"
Nov 02 22:03:25.030666 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:03:25.030646    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c\": container with ID starting with 64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c not found: ID does not exist" containerID="64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c"
Nov 02 22:03:25.030742 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.030673    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c"} err="failed to get container status \"64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c\": rpc error: code = NotFound desc = could not find container \"64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c\": container with ID starting with 64d35a6f4f569e2efaf1aafd4d426f6a67618c15ee6c108562c1c3cc1aa4382c not found: ID does not exist"
Nov 02 22:03:25.185260 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.185223    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"]
Nov 02 22:03:25.188550 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:25.188526    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/auth-test-model-97995b-kserve-74445cfd57-hndzz"]
Nov 02 22:03:26.913597 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:26.913564    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197d7371-cd9c-4fc4-959e-d319d38cf7dc" path="/var/lib/kubelet/pods/197d7371-cd9c-4fc4-959e-d319d38cf7dc/volumes"
Nov 02 22:03:30.112562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.112529    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-0694c7"]
Nov 02 22:03:30.113138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.113094    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bcb8d3a0-7eac-49d7-829f-50e5133987f1" containerName="revoke-keys"
Nov 02 22:03:30.113138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.113121    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb8d3a0-7eac-49d7-829f-50e5133987f1" containerName="revoke-keys"
Nov 02 22:03:30.113241 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.113163    4528 cpu_manager.go:414] "containerMap: removing container" podUID="197d7371-cd9c-4fc4-959e-d319d38cf7dc" containerName="main"
Nov 02 22:03:30.113241 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.113170    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197d7371-cd9c-4fc4-959e-d319d38cf7dc" containerName="main"
Nov 02 22:03:30.113316 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.113253    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="197d7371-cd9c-4fc4-959e-d319d38cf7dc" containerName="main"
Nov 02 22:03:30.113316 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.113265    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb8d3a0-7eac-49d7-829f-50e5133987f1" containerName="revoke-keys"
Nov 02 22:03:30.118229 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.118210    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-0694c7"
Nov 02 22:03:30.166269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.166244    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-0694c7"]
Nov 02 22:03:30.175309 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.175288    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bds\" (UniqueName: \"kubernetes.io/projected/d625ccf6-557e-416a-b768-d245fee87ba1-kube-api-access-56bds\") pod \"test-curl-7477-0694c7\" (UID: \"d625ccf6-557e-416a-b768-d245fee87ba1\") " pod="opendatahub/test-curl-7477-0694c7"
Nov 02 22:03:30.213212 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.213185    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5v2nf\""
Nov 02 22:03:30.275985 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.275952    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bds\" (UniqueName: \"kubernetes.io/projected/d625ccf6-557e-416a-b768-d245fee87ba1-kube-api-access-56bds\") pod \"test-curl-7477-0694c7\" (UID: \"d625ccf6-557e-416a-b768-d245fee87ba1\") " pod="opendatahub/test-curl-7477-0694c7"
Nov 02 22:03:30.332752 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.332721    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bds\" (UniqueName: \"kubernetes.io/projected/d625ccf6-557e-416a-b768-d245fee87ba1-kube-api-access-56bds\") pod \"test-curl-7477-0694c7\" (UID: \"d625ccf6-557e-416a-b768-d245fee87ba1\") " pod="opendatahub/test-curl-7477-0694c7"
Nov 02 22:03:30.430554 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.430481    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-0694c7"
Nov 02 22:03:30.805018 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:30.804956    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-0694c7"]
Nov 02 22:03:31.059339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:31.059249    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-0694c7" event={"ID":"d625ccf6-557e-416a-b768-d245fee87ba1","Type":"ContainerStarted","Data":"54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a"}
Nov 02 22:03:31.059339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:31.059296    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-0694c7" event={"ID":"d625ccf6-557e-416a-b768-d245fee87ba1","Type":"ContainerStarted","Data":"0b175de36f416c9d1bdda31e70541a8d40665d1e923cf554f2bc4257789df11b"}
Nov 02 22:03:31.171569 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:31.171522    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7477-0694c7" podStartSLOduration=1.171508775 podStartE2EDuration="1.171508775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:31.169468285 +0000 UTC m=+3298.433004603" watchObservedRunningTime="2026-11-02 22:03:31.171508775 +0000 UTC m=+3298.435045092"
Nov 02 22:03:32.624550 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:32.624514    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-0694c7"]
Nov 02 22:03:32.640825 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:32.640790    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-0694c7"]
Nov 02 22:03:33.071101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:33.071068    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7477-0694c7" podUID="d625ccf6-557e-416a-b768-d245fee87ba1" containerName="test-curl-7477-0694c7" containerID="cri-o://54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a" gracePeriod=2
Nov 02 22:03:35.220359 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.220326    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-3c9d29"]
Nov 02 22:03:35.221244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.221222    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d625ccf6-557e-416a-b768-d245fee87ba1" containerName="test-curl-7477-0694c7"
Nov 02 22:03:35.221244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.221244    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d625ccf6-557e-416a-b768-d245fee87ba1" containerName="test-curl-7477-0694c7"
Nov 02 22:03:35.221426 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.221392    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d625ccf6-557e-416a-b768-d245fee87ba1" containerName="test-curl-7477-0694c7"
Nov 02 22:03:35.224699 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.224680    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-3c9d29"
Nov 02 22:03:35.274637 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.274605    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-3c9d29"]
Nov 02 22:03:35.332415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.332350    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-0694c7"
Nov 02 22:03:35.346266 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.346240    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrgg\" (UniqueName: \"kubernetes.io/projected/ce1d12db-521d-4563-baa7-14e1fa58e269-kube-api-access-gcrgg\") pod \"test-curl-7477-3c9d29\" (UID: \"ce1d12db-521d-4563-baa7-14e1fa58e269\") " pod="opendatahub/test-curl-7477-3c9d29"
Nov 02 22:03:35.447778 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.447741    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d625ccf6-557e-416a-b768-d245fee87ba1-kube-api-access-56bds\" (UniqueName: \"kubernetes.io/projected/d625ccf6-557e-416a-b768-d245fee87ba1-kube-api-access-56bds\") pod \"d625ccf6-557e-416a-b768-d245fee87ba1\" (UID: \"d625ccf6-557e-416a-b768-d245fee87ba1\") "
Nov 02 22:03:35.448087 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.448069    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrgg\" (UniqueName: \"kubernetes.io/projected/ce1d12db-521d-4563-baa7-14e1fa58e269-kube-api-access-gcrgg\") pod \"test-curl-7477-3c9d29\" (UID: \"ce1d12db-521d-4563-baa7-14e1fa58e269\") " pod="opendatahub/test-curl-7477-3c9d29"
Nov 02 22:03:35.449899 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.449874    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d625ccf6-557e-416a-b768-d245fee87ba1-kube-api-access-56bds" pod "d625ccf6-557e-416a-b768-d245fee87ba1" (UID: "d625ccf6-557e-416a-b768-d245fee87ba1"). InnerVolumeSpecName "kube-api-access-56bds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:03:35.505569 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.505542    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrgg\" (UniqueName: \"kubernetes.io/projected/ce1d12db-521d-4563-baa7-14e1fa58e269-kube-api-access-gcrgg\") pod \"test-curl-7477-3c9d29\" (UID: \"ce1d12db-521d-4563-baa7-14e1fa58e269\") " pod="opendatahub/test-curl-7477-3c9d29"
Nov 02 22:03:35.537688 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.537661    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-3c9d29"
Nov 02 22:03:35.549384 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.549358    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bds\" (UniqueName: \"kubernetes.io/projected/d625ccf6-557e-416a-b768-d245fee87ba1-kube-api-access-56bds\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:35.906369 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:03:35.906339    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce1d12db_521d_4563_baa7_14e1fa58e269.slice/crio-684ca40f998b9baac2c57ab8c2eeea55e6ba982fa587865645ad94cd05184253 WatchSource:0}: Error finding container 684ca40f998b9baac2c57ab8c2eeea55e6ba982fa587865645ad94cd05184253: Status 404 returned error can't find the container with id 684ca40f998b9baac2c57ab8c2eeea55e6ba982fa587865645ad94cd05184253
Nov 02 22:03:35.909073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:35.909028    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-3c9d29"]
Nov 02 22:03:36.072809 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.072778    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"]
Nov 02 22:03:36.073021 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.073001    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy" containerID="cri-o://a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d" gracePeriod=30
Nov 02 22:03:36.095826 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.095790    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-3c9d29" event={"ID":"ce1d12db-521d-4563-baa7-14e1fa58e269","Type":"ContainerStarted","Data":"58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b"}
Nov 02 22:03:36.096011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.095839    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-3c9d29" event={"ID":"ce1d12db-521d-4563-baa7-14e1fa58e269","Type":"ContainerStarted","Data":"684ca40f998b9baac2c57ab8c2eeea55e6ba982fa587865645ad94cd05184253"}
Nov 02 22:03:36.097179 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.097154    4528 generic.go:356] "Generic (PLEG): container finished" podID="d625ccf6-557e-416a-b768-d245fee87ba1" containerID="54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a" exitCode=137
Nov 02 22:03:36.097329 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.097191    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-0694c7"
Nov 02 22:03:36.097329 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.097199    4528 scope.go:122] "RemoveContainer" containerID="54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a"
Nov 02 22:03:36.108154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.106594    4528 scope.go:122] "RemoveContainer" containerID="54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a"
Nov 02 22:03:36.108154 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:03:36.106988    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a\": container with ID starting with 54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a not found: ID does not exist" containerID="54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a"
Nov 02 22:03:36.108154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.107024    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a"} err="failed to get container status \"54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a\": rpc error: code = NotFound desc = could not find container \"54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a\": container with ID starting with 54e67c1b074933eb58a151d69b62b9078bc6bdffa2d7492c24879f161864877a not found: ID does not exist"
Nov 02 22:03:36.122163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.122121    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"]
Nov 02 22:03:36.122376 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.122352    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy" containerID="cri-o://59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f" gracePeriod=30
Nov 02 22:03:36.417841 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.417791    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7477-3c9d29" podStartSLOduration=1.417776851 podStartE2EDuration="1.417776851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:36.417601823 +0000 UTC m=+3303.681138140" watchObservedRunningTime="2026-11-02 22:03:36.417776851 +0000 UTC m=+3303.681313169"
Nov 02 22:03:36.468772 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:03:36.468732    4528 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7477-0694c7\" is forbidden: User \"system:node:ip-10-0-133-86.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-86.us-west-2.compute.internal' and this object" podUID="d625ccf6-557e-416a-b768-d245fee87ba1" pod="opendatahub/test-curl-7477-0694c7"
Nov 02 22:03:36.918092 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:36.918048    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d625ccf6-557e-416a-b768-d245fee87ba1" path="/var/lib/kubelet/pods/d625ccf6-557e-416a-b768-d245fee87ba1/volumes"
Nov 02 22:03:37.810767 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:37.810718    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-3c9d29"]
Nov 02 22:03:37.814781 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:37.814751    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-3c9d29"]
Nov 02 22:03:38.110096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:38.110005    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7477-3c9d29" podUID="ce1d12db-521d-4563-baa7-14e1fa58e269" containerName="test-curl-7477-3c9d29" containerID="cri-o://58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b" gracePeriod=2
Nov 02 22:03:38.161370 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:03:38.161334    4528 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7477-3c9d29\" is forbidden: User \"system:node:ip-10-0-133-86.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-86.us-west-2.compute.internal' and this object" podUID="ce1d12db-521d-4563-baa7-14e1fa58e269" pod="opendatahub/test-curl-7477-3c9d29"
Nov 02 22:03:39.472723 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:39.472678    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:03:39.473184 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:39.472742    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:03:40.171724 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:40.171692    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:03:40.171860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:40.171755    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:03:40.356099 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:40.356071    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-3c9d29"
Nov 02 22:03:40.436400 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:40.436318    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1d12db-521d-4563-baa7-14e1fa58e269-kube-api-access-gcrgg\" (UniqueName: \"kubernetes.io/projected/ce1d12db-521d-4563-baa7-14e1fa58e269-kube-api-access-gcrgg\") pod \"ce1d12db-521d-4563-baa7-14e1fa58e269\" (UID: \"ce1d12db-521d-4563-baa7-14e1fa58e269\") "
Nov 02 22:03:40.438454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:40.438430    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1d12db-521d-4563-baa7-14e1fa58e269-kube-api-access-gcrgg" pod "ce1d12db-521d-4563-baa7-14e1fa58e269" (UID: "ce1d12db-521d-4563-baa7-14e1fa58e269"). InnerVolumeSpecName "kube-api-access-gcrgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:03:40.538223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:40.538191    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcrgg\" (UniqueName: \"kubernetes.io/projected/ce1d12db-521d-4563-baa7-14e1fa58e269-kube-api-access-gcrgg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:40.936070 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:40.934939    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1d12db-521d-4563-baa7-14e1fa58e269" path="/var/lib/kubelet/pods/ce1d12db-521d-4563-baa7-14e1fa58e269/volumes"
Nov 02 22:03:41.131684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.131652    4528 generic.go:356] "Generic (PLEG): container finished" podID="ce1d12db-521d-4563-baa7-14e1fa58e269" containerID="58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b" exitCode=137
Nov 02 22:03:41.131848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.131701    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-3c9d29"
Nov 02 22:03:41.131848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.131719    4528 scope.go:122] "RemoveContainer" containerID="58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b"
Nov 02 22:03:41.217914 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.217891    4528 scope.go:122] "RemoveContainer" containerID="58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b"
Nov 02 22:03:41.218292 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:03:41.218270    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b\": container with ID starting with 58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b not found: ID does not exist" containerID="58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b"
Nov 02 22:03:41.218341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.218302    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b"} err="failed to get container status \"58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b\": rpc error: code = NotFound desc = could not find container \"58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b\": container with ID starting with 58361c4f01f55d9997ee6dd519476b0b3ffdf7f743a9f0f77299e127a72b306b not found: ID does not exist"
Nov 02 22:03:41.394042 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.394015    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 22:03:41.427852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.427827    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 22:03:41.549440 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549410    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-data\") pod \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") "
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549466    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istiod-ca-cert\") pod \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") "
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549505    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-podinfo\") pod \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") "
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549539    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-envoy\") pod \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") "
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549559    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-data\") pod \"c0524738-f680-4ece-a83d-61b302af5aff\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") "
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549616    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-credential-socket\") pod \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") "
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549654    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-kube-api-access-xdpbz\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-kube-api-access-xdpbz\") pod \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") "
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549696    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-certs\") pod \"c0524738-f680-4ece-a83d-61b302af5aff\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") "
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549741    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-data" pod "49db25d8-09d9-45be-8fc7-5482fb7a9fbb" (UID: "49db25d8-09d9-45be-8fc7-5482fb7a9fbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549749    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-token\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-token\") pod \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") "
Nov 02 22:03:41.549888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549882    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-envoy\") pod \"c0524738-f680-4ece-a83d-61b302af5aff\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") "
Nov 02 22:03:41.550386 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549918    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-data" pod "c0524738-f680-4ece-a83d-61b302af5aff" (UID: "c0524738-f680-4ece-a83d-61b302af5aff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.550386 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.549934    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-certs\") pod \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") "
Nov 02 22:03:41.550623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.550594    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-credential-socket" pod "49db25d8-09d9-45be-8fc7-5482fb7a9fbb" (UID: "49db25d8-09d9-45be-8fc7-5482fb7a9fbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.550743 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.550718    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-certs" pod "c0524738-f680-4ece-a83d-61b302af5aff" (UID: "c0524738-f680-4ece-a83d-61b302af5aff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.550945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.550910    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-certs" pod "49db25d8-09d9-45be-8fc7-5482fb7a9fbb" (UID: "49db25d8-09d9-45be-8fc7-5482fb7a9fbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.551254 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.551232    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istiod-ca-cert" pod "49db25d8-09d9-45be-8fc7-5482fb7a9fbb" (UID: "49db25d8-09d9-45be-8fc7-5482fb7a9fbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:03:41.551335 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.551279    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-credential-socket\") pod \"c0524738-f680-4ece-a83d-61b302af5aff\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") "
Nov 02 22:03:41.551388 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.551366    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0524738-f680-4ece-a83d-61b302af5aff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0524738-f680-4ece-a83d-61b302af5aff-istio-podinfo\") pod \"c0524738-f680-4ece-a83d-61b302af5aff\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") "
Nov 02 22:03:41.551454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.551406    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-socket\") pod \"c0524738-f680-4ece-a83d-61b302af5aff\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") "
Nov 02 22:03:41.551512 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.551478    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-istio-token\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-istio-token\") pod \"c0524738-f680-4ece-a83d-61b302af5aff\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") "
Nov 02 22:03:41.551563 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.551520    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-socket\") pod \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\" (UID: \"49db25d8-09d9-45be-8fc7-5482fb7a9fbb\") "
Nov 02 22:03:41.551622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.551562    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-kube-api-access-znt7g\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-kube-api-access-znt7g\") pod \"c0524738-f680-4ece-a83d-61b302af5aff\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") "
Nov 02 22:03:41.551674 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.551624    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0524738-f680-4ece-a83d-61b302af5aff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0524738-f680-4ece-a83d-61b302af5aff-istiod-ca-cert\") pod \"c0524738-f680-4ece-a83d-61b302af5aff\" (UID: \"c0524738-f680-4ece-a83d-61b302af5aff\") "
Nov 02 22:03:41.552157 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.552123    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-socket" pod "c0524738-f680-4ece-a83d-61b302af5aff" (UID: "c0524738-f680-4ece-a83d-61b302af5aff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.552292 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.552222    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-socket" pod "49db25d8-09d9-45be-8fc7-5482fb7a9fbb" (UID: "49db25d8-09d9-45be-8fc7-5482fb7a9fbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.552755 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.552442    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-credential-socket" pod "c0524738-f680-4ece-a83d-61b302af5aff" (UID: "c0524738-f680-4ece-a83d-61b302af5aff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.553325 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.553297    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-kube-api-access-xdpbz" pod "49db25d8-09d9-45be-8fc7-5482fb7a9fbb" (UID: "49db25d8-09d9-45be-8fc7-5482fb7a9fbb"). InnerVolumeSpecName "kube-api-access-xdpbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.553668    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0524738-f680-4ece-a83d-61b302af5aff-istiod-ca-cert" pod "c0524738-f680-4ece-a83d-61b302af5aff" (UID: "c0524738-f680-4ece-a83d-61b302af5aff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554211    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-envoy" pod "49db25d8-09d9-45be-8fc7-5482fb7a9fbb" (UID: "49db25d8-09d9-45be-8fc7-5482fb7a9fbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554280    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-token" pod "49db25d8-09d9-45be-8fc7-5482fb7a9fbb" (UID: "49db25d8-09d9-45be-8fc7-5482fb7a9fbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554491    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554511    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554509    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-podinfo" pod "49db25d8-09d9-45be-8fc7-5482fb7a9fbb" (UID: "49db25d8-09d9-45be-8fc7-5482fb7a9fbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554532    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0524738-f680-4ece-a83d-61b302af5aff-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554548    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554562    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554575    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554595    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554610    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdpbz\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-kube-api-access-xdpbz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554625    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554640    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.554955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.554659    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.555762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.555175    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-istio-token" pod "c0524738-f680-4ece-a83d-61b302af5aff" (UID: "c0524738-f680-4ece-a83d-61b302af5aff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:03:41.555762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.555260    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-kube-api-access-znt7g" pod "c0524738-f680-4ece-a83d-61b302af5aff" (UID: "c0524738-f680-4ece-a83d-61b302af5aff"). InnerVolumeSpecName "kube-api-access-znt7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:03:41.555762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.555426    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0524738-f680-4ece-a83d-61b302af5aff-istio-podinfo" pod "c0524738-f680-4ece-a83d-61b302af5aff" (UID: "c0524738-f680-4ece-a83d-61b302af5aff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:03:41.567431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.567358    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-envoy" pod "c0524738-f680-4ece-a83d-61b302af5aff" (UID: "c0524738-f680-4ece-a83d-61b302af5aff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:03:41.656308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.656272    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.656308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.656301    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.656308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.656310    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49db25d8-09d9-45be-8fc7-5482fb7a9fbb-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.656308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.656321    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0524738-f680-4ece-a83d-61b302af5aff-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.656600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.656329    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0524738-f680-4ece-a83d-61b302af5aff-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.656600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.656338    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:41.656600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:41.656348    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znt7g\" (UniqueName: \"kubernetes.io/projected/c0524738-f680-4ece-a83d-61b302af5aff-kube-api-access-znt7g\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:42.140140 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.140084    4528 generic.go:356] "Generic (PLEG): container finished" podID="c0524738-f680-4ece-a83d-61b302af5aff" containerID="a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d" exitCode=0
Nov 02 22:03:42.140360 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.140193    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"
Nov 02 22:03:42.140360 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.140197    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" event={"ID":"c0524738-f680-4ece-a83d-61b302af5aff","Type":"ContainerDied","Data":"a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d"}
Nov 02 22:03:42.140360 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.140290    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd" event={"ID":"c0524738-f680-4ece-a83d-61b302af5aff","Type":"ContainerDied","Data":"ffdafeba81ab572d5817cddd09abf6321007f371ed33908468370888dad378ad"}
Nov 02 22:03:42.140360 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.140325    4528 scope.go:122] "RemoveContainer" containerID="a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d"
Nov 02 22:03:42.142095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.141658    4528 generic.go:356] "Generic (PLEG): container finished" podID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerID="59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f" exitCode=0
Nov 02 22:03:42.142095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.141725    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"
Nov 02 22:03:42.142095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.141748    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" event={"ID":"49db25d8-09d9-45be-8fc7-5482fb7a9fbb","Type":"ContainerDied","Data":"59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f"}
Nov 02 22:03:42.142095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.141809    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb" event={"ID":"49db25d8-09d9-45be-8fc7-5482fb7a9fbb","Type":"ContainerDied","Data":"c59e99175a6f342ed6641acf5c5037eec1f6d511cbbd0f9de814640e16e52e72"}
Nov 02 22:03:42.152716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.152695    4528 scope.go:122] "RemoveContainer" containerID="a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d"
Nov 02 22:03:42.153018 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:03:42.152994    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d\": container with ID starting with a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d not found: ID does not exist" containerID="a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d"
Nov 02 22:03:42.153144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.153023    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d"} err="failed to get container status \"a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d\": rpc error: code = NotFound desc = could not find container \"a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d\": container with ID starting with a341368b96693edc44137860d41e3d70c47a7a112bd1f127d2b17672a2c4a78d not found: ID does not exist"
Nov 02 22:03:42.153144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.153040    4528 scope.go:122] "RemoveContainer" containerID="59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f"
Nov 02 22:03:42.171253 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.171233    4528 scope.go:122] "RemoveContainer" containerID="59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f"
Nov 02 22:03:42.171538 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:03:42.171521    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f\": container with ID starting with 59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f not found: ID does not exist" containerID="59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f"
Nov 02 22:03:42.171588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.171544    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f"} err="failed to get container status \"59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f\": rpc error: code = NotFound desc = could not find container \"59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f\": container with ID starting with 59ffd133317570ae4d7dbf453ae5eea6003396701d1f3f1883ad6bc38409f76f not found: ID does not exist"
Nov 02 22:03:42.305726 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.305693    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"]
Nov 02 22:03:42.355966 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.355930    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-mmcnd"]
Nov 02 22:03:42.416100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.416016    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"]
Nov 02 22:03:42.419865 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.419839    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-zf5vb"]
Nov 02 22:03:42.913998 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.913966    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" path="/var/lib/kubelet/pods/49db25d8-09d9-45be-8fc7-5482fb7a9fbb/volumes"
Nov 02 22:03:42.914439 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:42.914423    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0524738-f680-4ece-a83d-61b302af5aff" path="/var/lib/kubelet/pods/c0524738-f680-4ece-a83d-61b302af5aff/volumes"
Nov 02 22:03:43.806009 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:43.805979    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:03:44.114459 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:03:44.114362    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:03:47.143943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.143910    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"]
Nov 02 22:03:47.144565 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.144548    4528 cpu_manager.go:414] "containerMap: removing container" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy"
Nov 02 22:03:47.144626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.144567    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy"
Nov 02 22:03:47.144626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.144603    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy"
Nov 02 22:03:47.144626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.144608    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy"
Nov 02 22:03:47.144626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.144624    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ce1d12db-521d-4563-baa7-14e1fa58e269" containerName="test-curl-7477-3c9d29"
Nov 02 22:03:47.144784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.144630    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1d12db-521d-4563-baa7-14e1fa58e269" containerName="test-curl-7477-3c9d29"
Nov 02 22:03:47.144784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.144727    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="49db25d8-09d9-45be-8fc7-5482fb7a9fbb" containerName="istio-proxy"
Nov 02 22:03:47.144784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.144745    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0524738-f680-4ece-a83d-61b302af5aff" containerName="istio-proxy"
Nov 02 22:03:47.144784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.144755    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1d12db-521d-4563-baa7-14e1fa58e269" containerName="test-curl-7477-3c9d29"
Nov 02 22:03:47.149834 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.149816    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:03:47.153882 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.153857    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9366e9-80af-4bc3-a179-d18ea766eab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44\" (UID: \"bb9366e9-80af-4bc3-a179-d18ea766eab3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:03:47.154013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.153890    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllwt\" (UniqueName: \"kubernetes.io/projected/bb9366e9-80af-4bc3-a179-d18ea766eab3-kube-api-access-qllwt\") pod \"payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44\" (UID: \"bb9366e9-80af-4bc3-a179-d18ea766eab3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:03:47.199942 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.199905    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"]
Nov 02 22:03:47.247634 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.247582    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8702a9-dockercfg-mrhm7\""
Nov 02 22:03:47.247634 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.247602    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8702a9\""
Nov 02 22:03:47.272181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.272138    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9366e9-80af-4bc3-a179-d18ea766eab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44\" (UID: \"bb9366e9-80af-4bc3-a179-d18ea766eab3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:03:47.276882 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.276845    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllwt\" (UniqueName: \"kubernetes.io/projected/bb9366e9-80af-4bc3-a179-d18ea766eab3-kube-api-access-qllwt\") pod \"payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44\" (UID: \"bb9366e9-80af-4bc3-a179-d18ea766eab3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:03:47.279696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.276671    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9366e9-80af-4bc3-a179-d18ea766eab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44\" (UID: \"bb9366e9-80af-4bc3-a179-d18ea766eab3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:03:47.306804 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.306762    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"]
Nov 02 22:03:47.313445 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.313415    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:47.342175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.340957    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllwt\" (UniqueName: \"kubernetes.io/projected/bb9366e9-80af-4bc3-a179-d18ea766eab3-kube-api-access-qllwt\") pod \"payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44\" (UID: \"bb9366e9-80af-4bc3-a179-d18ea766eab3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:03:47.362607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.362575    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"]
Nov 02 22:03:47.380687 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.380654    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnqc7\" (UniqueName: \"kubernetes.io/projected/d95af97f-15a0-4fc3-9804-e5a0b54e807d-kube-api-access-fnqc7\") pod \"payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz\" (UID: \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:47.380891 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.380830    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95af97f-15a0-4fc3-9804-e5a0b54e807d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz\" (UID: \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:47.463242 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.463153    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:03:47.481854 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.481817    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95af97f-15a0-4fc3-9804-e5a0b54e807d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz\" (UID: \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:47.481993 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.481973    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqc7\" (UniqueName: \"kubernetes.io/projected/d95af97f-15a0-4fc3-9804-e5a0b54e807d-kube-api-access-fnqc7\") pod \"payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz\" (UID: \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:47.482543 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.482523    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95af97f-15a0-4fc3-9804-e5a0b54e807d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz\" (UID: \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:47.544043 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.544006    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqc7\" (UniqueName: \"kubernetes.io/projected/d95af97f-15a0-4fc3-9804-e5a0b54e807d-kube-api-access-fnqc7\") pod \"payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz\" (UID: \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:47.658321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.658286    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:47.835535 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:03:47.835496    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9366e9_80af_4bc3_a179_d18ea766eab3.slice/crio-5e83491e1c01654c798ac0f619969ed9abce8868f07b5b38cec1b29072b0738d WatchSource:0}: Error finding container 5e83491e1c01654c798ac0f619969ed9abce8868f07b5b38cec1b29072b0738d: Status 404 returned error can't find the container with id 5e83491e1c01654c798ac0f619969ed9abce8868f07b5b38cec1b29072b0738d
Nov 02 22:03:47.838510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:47.838486    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"]
Nov 02 22:03:48.027016 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:03:48.026982    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95af97f_15a0_4fc3_9804_e5a0b54e807d.slice/crio-e10a6d25ddb8aa4c611c30491b424121df7ebcb74895704ed43605c2c0e57ff0 WatchSource:0}: Error finding container e10a6d25ddb8aa4c611c30491b424121df7ebcb74895704ed43605c2c0e57ff0: Status 404 returned error can't find the container with id e10a6d25ddb8aa4c611c30491b424121df7ebcb74895704ed43605c2c0e57ff0
Nov 02 22:03:48.029719 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:48.029690    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"]
Nov 02 22:03:48.183658 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:48.183625    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44" event={"ID":"bb9366e9-80af-4bc3-a179-d18ea766eab3","Type":"ContainerStarted","Data":"d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766"}
Nov 02 22:03:48.184101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:48.183682    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44" event={"ID":"bb9366e9-80af-4bc3-a179-d18ea766eab3","Type":"ContainerStarted","Data":"5e83491e1c01654c798ac0f619969ed9abce8868f07b5b38cec1b29072b0738d"}
Nov 02 22:03:48.184101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:48.183732    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:03:48.186072 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:48.186047    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz" event={"ID":"d95af97f-15a0-4fc3-9804-e5a0b54e807d","Type":"ContainerStarted","Data":"3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b"}
Nov 02 22:03:48.186235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:48.186087    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz" event={"ID":"d95af97f-15a0-4fc3-9804-e5a0b54e807d","Type":"ContainerStarted","Data":"e10a6d25ddb8aa4c611c30491b424121df7ebcb74895704ed43605c2c0e57ff0"}
Nov 02 22:03:48.186235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:48.186143    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:48.299650 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:48.299549    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44" podStartSLOduration=1.29953287 podStartE2EDuration="1.29953287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:48.296495793 +0000 UTC m=+3315.560032110" watchObservedRunningTime="2026-11-02 22:03:48.29953287 +0000 UTC m=+3315.563069187"
Nov 02 22:03:48.430600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:48.429706    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz" podStartSLOduration=1.429688561 podStartE2EDuration="1.429688561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:48.427026896 +0000 UTC m=+3315.690563215" watchObservedRunningTime="2026-11-02 22:03:48.429688561 +0000 UTC m=+3315.693224880"
Nov 02 22:03:52.387464 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.387423    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"]
Nov 02 22:03:52.391850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.391828    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.439118 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.439070    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c900efb-hfz7t\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.439280 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.439190    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxbt6\" (UniqueName: \"kubernetes.io/projected/bb4e238e-ab7f-438d-9684-06fce8976d49-kube-api-access-vxbt6\") pod \"maas-api-revoke-sub-2c900efb-hfz7t\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.439349 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.439305    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c900efb-hfz7t\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.443896 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.443873    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"]
Nov 02 22:03:52.490456 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.490424    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:03:52.490456 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.490435    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:03:52.490672 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.490510    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:03:52.495830 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.495817    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:03:52.540506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.540479    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c900efb-hfz7t\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.540611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.540528    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxbt6\" (UniqueName: \"kubernetes.io/projected/bb4e238e-ab7f-438d-9684-06fce8976d49-kube-api-access-vxbt6\") pod \"maas-api-revoke-sub-2c900efb-hfz7t\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.540611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.540595    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c900efb-hfz7t\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.541400 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.541380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c900efb-hfz7t\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.542859 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.542842    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c900efb-hfz7t\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.597309 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.597279    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxbt6\" (UniqueName: \"kubernetes.io/projected/bb4e238e-ab7f-438d-9684-06fce8976d49-kube-api-access-vxbt6\") pod \"maas-api-revoke-sub-2c900efb-hfz7t\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:52.705921 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:52.705844    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:53.076741 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:03:53.076711    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4e238e_ab7f_438d_9684_06fce8976d49.slice/crio-72089fedd509eaf4155c0dc635a176a808b314168060a782e7e3d4443a5ddc0d WatchSource:0}: Error finding container 72089fedd509eaf4155c0dc635a176a808b314168060a782e7e3d4443a5ddc0d: Status 404 returned error can't find the container with id 72089fedd509eaf4155c0dc635a176a808b314168060a782e7e3d4443a5ddc0d
Nov 02 22:03:53.080244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:53.080219    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"]
Nov 02 22:03:53.217898 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:53.217861    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t" event={"ID":"bb4e238e-ab7f-438d-9684-06fce8976d49","Type":"ContainerStarted","Data":"72089fedd509eaf4155c0dc635a176a808b314168060a782e7e3d4443a5ddc0d"}
Nov 02 22:03:54.225918 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:54.225841    4528 generic.go:356] "Generic (PLEG): container finished" podID="bb4e238e-ab7f-438d-9684-06fce8976d49" containerID="465e890fb66fb22f902a82522c010c5bd338fef499de695eb83536c46bbe795d" exitCode=0
Nov 02 22:03:54.225918 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:54.225898    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t" event={"ID":"bb4e238e-ab7f-438d-9684-06fce8976d49","Type":"ContainerDied","Data":"465e890fb66fb22f902a82522c010c5bd338fef499de695eb83536c46bbe795d"}
Nov 02 22:03:55.375612 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.375587    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:55.487349 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.487272    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-openshift-ca\") pod \"bb4e238e-ab7f-438d-9684-06fce8976d49\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") "
Nov 02 22:03:55.487528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.487377    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4e238e-ab7f-438d-9684-06fce8976d49-kube-api-access-vxbt6\" (UniqueName: \"kubernetes.io/projected/bb4e238e-ab7f-438d-9684-06fce8976d49-kube-api-access-vxbt6\") pod \"bb4e238e-ab7f-438d-9684-06fce8976d49\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") "
Nov 02 22:03:55.487528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.487507    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-xks-ca\") pod \"bb4e238e-ab7f-438d-9684-06fce8976d49\" (UID: \"bb4e238e-ab7f-438d-9684-06fce8976d49\") "
Nov 02 22:03:55.487717 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.487689    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-openshift-ca" pod "bb4e238e-ab7f-438d-9684-06fce8976d49" (UID: "bb4e238e-ab7f-438d-9684-06fce8976d49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:03:55.488027 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.488001    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:55.489855 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.489826    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-xks-ca" pod "bb4e238e-ab7f-438d-9684-06fce8976d49" (UID: "bb4e238e-ab7f-438d-9684-06fce8976d49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:03:55.490078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.490062    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4e238e-ab7f-438d-9684-06fce8976d49-kube-api-access-vxbt6" pod "bb4e238e-ab7f-438d-9684-06fce8976d49" (UID: "bb4e238e-ab7f-438d-9684-06fce8976d49"). InnerVolumeSpecName "kube-api-access-vxbt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:03:55.589201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.589166    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxbt6\" (UniqueName: \"kubernetes.io/projected/bb4e238e-ab7f-438d-9684-06fce8976d49-kube-api-access-vxbt6\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:55.589201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:55.589195    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4e238e-ab7f-438d-9684-06fce8976d49-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:03:56.240221 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:56.240186    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"
Nov 02 22:03:56.240221 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:56.240203    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t" event={"ID":"bb4e238e-ab7f-438d-9684-06fce8976d49","Type":"ContainerDied","Data":"72089fedd509eaf4155c0dc635a176a808b314168060a782e7e3d4443a5ddc0d"}
Nov 02 22:03:56.240447 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:56.240263    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72089fedd509eaf4155c0dc635a176a808b314168060a782e7e3d4443a5ddc0d"
Nov 02 22:03:59.195624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:59.195593    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:03:59.196024 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:03:59.195668    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:04:03.158366 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.158331    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"]
Nov 02 22:04:03.158949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.158936    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bb4e238e-ab7f-438d-9684-06fce8976d49" containerName="revoke-keys"
Nov 02 22:04:03.158999 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.158950    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4e238e-ab7f-438d-9684-06fce8976d49" containerName="revoke-keys"
Nov 02 22:04:03.159043 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.159037    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4e238e-ab7f-438d-9684-06fce8976d49" containerName="revoke-keys"
Nov 02 22:04:03.162699 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.162681    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.219194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.219160    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"]
Nov 02 22:04:03.261541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.261508    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:04:03.261541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.261527    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:04:03.261541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.261515    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:04:03.261802 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.261592    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:04:03.285345 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.285309    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34604d88-db75-4517-9c28-8288012cbd92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6df04840-xcldj\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.285532 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.285441    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34604d88-db75-4517-9c28-8288012cbd92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6df04840-xcldj\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.285532 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.285462    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl2zt\" (UniqueName: \"kubernetes.io/projected/34604d88-db75-4517-9c28-8288012cbd92-kube-api-access-zl2zt\") pod \"maas-api-revoke-sub-6df04840-xcldj\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.386180 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.386141    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34604d88-db75-4517-9c28-8288012cbd92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6df04840-xcldj\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.386402 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.386256    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34604d88-db75-4517-9c28-8288012cbd92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6df04840-xcldj\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.386402 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.386284    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl2zt\" (UniqueName: \"kubernetes.io/projected/34604d88-db75-4517-9c28-8288012cbd92-kube-api-access-zl2zt\") pod \"maas-api-revoke-sub-6df04840-xcldj\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.386874 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.386843    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34604d88-db75-4517-9c28-8288012cbd92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6df04840-xcldj\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.388576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.388556    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34604d88-db75-4517-9c28-8288012cbd92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6df04840-xcldj\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.446230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.446093    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl2zt\" (UniqueName: \"kubernetes.io/projected/34604d88-db75-4517-9c28-8288012cbd92-kube-api-access-zl2zt\") pod \"maas-api-revoke-sub-6df04840-xcldj\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.476529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.476497    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:03.850883 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:04:03.850849    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34604d88_db75_4517_9c28_8288012cbd92.slice/crio-e176b0d0a78e83a91e093b99511108b3635151712f7b388b516c9c091a3b52d1 WatchSource:0}: Error finding container e176b0d0a78e83a91e093b99511108b3635151712f7b388b516c9c091a3b52d1: Status 404 returned error can't find the container with id e176b0d0a78e83a91e093b99511108b3635151712f7b388b516c9c091a3b52d1
Nov 02 22:04:03.853166 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:03.853143    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"]
Nov 02 22:04:04.293569 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:04.293529    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj" event={"ID":"34604d88-db75-4517-9c28-8288012cbd92","Type":"ContainerStarted","Data":"9ce54752be69d02ce8b0c799fb5ac2ccb87effdd13d5ee0511b520f58c10e138"}
Nov 02 22:04:04.294021 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:04.293590    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj" event={"ID":"34604d88-db75-4517-9c28-8288012cbd92","Type":"ContainerStarted","Data":"e176b0d0a78e83a91e093b99511108b3635151712f7b388b516c9c091a3b52d1"}
Nov 02 22:04:04.410539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:04.410442    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj" podStartSLOduration=1.410427299 podStartE2EDuration="1.410427299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:04.407547439 +0000 UTC m=+3331.671083772" watchObservedRunningTime="2026-11-02 22:04:04.410427299 +0000 UTC m=+3331.673963624"
Nov 02 22:04:05.302306 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:05.302272    4528 generic.go:356] "Generic (PLEG): container finished" podID="34604d88-db75-4517-9c28-8288012cbd92" containerID="9ce54752be69d02ce8b0c799fb5ac2ccb87effdd13d5ee0511b520f58c10e138" exitCode=0
Nov 02 22:04:05.302746 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:05.302361    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj" event={"ID":"34604d88-db75-4517-9c28-8288012cbd92","Type":"ContainerDied","Data":"9ce54752be69d02ce8b0c799fb5ac2ccb87effdd13d5ee0511b520f58c10e138"}
Nov 02 22:04:06.469001 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.468967    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:06.623410 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.623315    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34604d88-db75-4517-9c28-8288012cbd92-kube-api-access-zl2zt\" (UniqueName: \"kubernetes.io/projected/34604d88-db75-4517-9c28-8288012cbd92-kube-api-access-zl2zt\") pod \"34604d88-db75-4517-9c28-8288012cbd92\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") "
Nov 02 22:04:06.623410 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.623372    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34604d88-db75-4517-9c28-8288012cbd92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34604d88-db75-4517-9c28-8288012cbd92-maas-api-xks-ca\") pod \"34604d88-db75-4517-9c28-8288012cbd92\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") "
Nov 02 22:04:06.623679 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.623452    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34604d88-db75-4517-9c28-8288012cbd92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34604d88-db75-4517-9c28-8288012cbd92-maas-api-openshift-ca\") pod \"34604d88-db75-4517-9c28-8288012cbd92\" (UID: \"34604d88-db75-4517-9c28-8288012cbd92\") "
Nov 02 22:04:06.623833 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.623800    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34604d88-db75-4517-9c28-8288012cbd92-maas-api-openshift-ca" pod "34604d88-db75-4517-9c28-8288012cbd92" (UID: "34604d88-db75-4517-9c28-8288012cbd92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:04:06.625575 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.625550    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34604d88-db75-4517-9c28-8288012cbd92-maas-api-xks-ca" pod "34604d88-db75-4517-9c28-8288012cbd92" (UID: "34604d88-db75-4517-9c28-8288012cbd92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:04:06.625684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.625630    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34604d88-db75-4517-9c28-8288012cbd92-kube-api-access-zl2zt" pod "34604d88-db75-4517-9c28-8288012cbd92" (UID: "34604d88-db75-4517-9c28-8288012cbd92"). InnerVolumeSpecName "kube-api-access-zl2zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:04:06.724711 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.724670    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34604d88-db75-4517-9c28-8288012cbd92-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:06.724711 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.724705    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl2zt\" (UniqueName: \"kubernetes.io/projected/34604d88-db75-4517-9c28-8288012cbd92-kube-api-access-zl2zt\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:06.724711 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:06.724719    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34604d88-db75-4517-9c28-8288012cbd92-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:07.318831 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:07.318799    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"
Nov 02 22:04:07.318831 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:07.318810    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj" event={"ID":"34604d88-db75-4517-9c28-8288012cbd92","Type":"ContainerDied","Data":"e176b0d0a78e83a91e093b99511108b3635151712f7b388b516c9c091a3b52d1"}
Nov 02 22:04:07.319090 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:07.318864    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e176b0d0a78e83a91e093b99511108b3635151712f7b388b516c9c091a3b52d1"
Nov 02 22:04:14.114601 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:04:14.114564    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:04:44.114203 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:04:44.114159    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:04:44.524022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:44.523992    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:04:52.683310 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.683276    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"]
Nov 02 22:04:52.685710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.683857    4528 cpu_manager.go:414] "containerMap: removing container" podUID="34604d88-db75-4517-9c28-8288012cbd92" containerName="revoke-keys"
Nov 02 22:04:52.685710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.683869    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34604d88-db75-4517-9c28-8288012cbd92" containerName="revoke-keys"
Nov 02 22:04:52.685710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.683970    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="34604d88-db75-4517-9c28-8288012cbd92" containerName="revoke-keys"
Nov 02 22:04:52.687284 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.687268    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:52.739056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.739020    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"]
Nov 02 22:04:52.739768 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.739737    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:04:52.739910 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.739818    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:04:52.739910 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.739835    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:04:52.784895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.784859    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77bc\" (UniqueName: \"kubernetes.io/projected/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-kube-api-access-t77bc\") pod \"maas-api-revoke-sub-3785705d-l4fhj\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:52.785095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.784936    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3785705d-l4fhj\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:52.785095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.785051    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3785705d-l4fhj\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:52.786210 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.786196    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:04:52.886487 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.886450    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77bc\" (UniqueName: \"kubernetes.io/projected/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-kube-api-access-t77bc\") pod \"maas-api-revoke-sub-3785705d-l4fhj\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:52.886685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.886521    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3785705d-l4fhj\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:52.886685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.886573    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3785705d-l4fhj\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:52.887182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.887163    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3785705d-l4fhj\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:52.888895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.888874    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3785705d-l4fhj\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:52.943747 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:52.943671    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77bc\" (UniqueName: \"kubernetes.io/projected/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-kube-api-access-t77bc\") pod \"maas-api-revoke-sub-3785705d-l4fhj\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:53.000519 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:53.000486    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:53.374223 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:04:53.374179    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907c8310_2e6e_4ce4_8a52_0c08a77ff0c8.slice/crio-676dedc1cce163ee4d51250cfc17246d7122ac04258d7265e6ec7ce320c45db5 WatchSource:0}: Error finding container 676dedc1cce163ee4d51250cfc17246d7122ac04258d7265e6ec7ce320c45db5: Status 404 returned error can't find the container with id 676dedc1cce163ee4d51250cfc17246d7122ac04258d7265e6ec7ce320c45db5
Nov 02 22:04:53.376940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:53.376911    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"]
Nov 02 22:04:53.649512 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:53.649480    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj" event={"ID":"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8","Type":"ContainerStarted","Data":"676dedc1cce163ee4d51250cfc17246d7122ac04258d7265e6ec7ce320c45db5"}
Nov 02 22:04:54.656789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:54.656697    4528 generic.go:356] "Generic (PLEG): container finished" podID="907c8310-2e6e-4ce4-8a52-0c08a77ff0c8" containerID="4819f50dea9e84a14e48ad5d55b7919334524b44be02c818a702a1a2f908d40d" exitCode=0
Nov 02 22:04:54.657174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:54.656781    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj" event={"ID":"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8","Type":"ContainerDied","Data":"4819f50dea9e84a14e48ad5d55b7919334524b44be02c818a702a1a2f908d40d"}
Nov 02 22:04:55.810906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:55.810881    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:55.919105 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:55.918999    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-xks-ca\") pod \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") "
Nov 02 22:04:55.919375 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:55.919196    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-kube-api-access-t77bc\" (UniqueName: \"kubernetes.io/projected/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-kube-api-access-t77bc\") pod \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") "
Nov 02 22:04:55.919375 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:55.919260    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-openshift-ca\") pod \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\" (UID: \"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8\") "
Nov 02 22:04:55.919607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:55.919582    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-openshift-ca" pod "907c8310-2e6e-4ce4-8a52-0c08a77ff0c8" (UID: "907c8310-2e6e-4ce4-8a52-0c08a77ff0c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:04:55.921311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:55.921280    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-xks-ca" pod "907c8310-2e6e-4ce4-8a52-0c08a77ff0c8" (UID: "907c8310-2e6e-4ce4-8a52-0c08a77ff0c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:04:55.921611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:55.921329    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-kube-api-access-t77bc" pod "907c8310-2e6e-4ce4-8a52-0c08a77ff0c8" (UID: "907c8310-2e6e-4ce4-8a52-0c08a77ff0c8"). InnerVolumeSpecName "kube-api-access-t77bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:04:56.021235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:56.021195    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77bc\" (UniqueName: \"kubernetes.io/projected/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-kube-api-access-t77bc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:56.021235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:56.021229    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:56.021470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:56.021246    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:04:56.669176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:56.669147    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"
Nov 02 22:04:56.669176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:56.669159    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj" event={"ID":"907c8310-2e6e-4ce4-8a52-0c08a77ff0c8","Type":"ContainerDied","Data":"676dedc1cce163ee4d51250cfc17246d7122ac04258d7265e6ec7ce320c45db5"}
Nov 02 22:04:56.669390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:04:56.669205    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676dedc1cce163ee4d51250cfc17246d7122ac04258d7265e6ec7ce320c45db5"
Nov 02 22:05:03.450985 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.450944    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"]
Nov 02 22:05:03.452440 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.452411    4528 cpu_manager.go:414] "containerMap: removing container" podUID="907c8310-2e6e-4ce4-8a52-0c08a77ff0c8" containerName="revoke-keys"
Nov 02 22:05:03.452440 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.452440    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907c8310-2e6e-4ce4-8a52-0c08a77ff0c8" containerName="revoke-keys"
Nov 02 22:05:03.452819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.452801    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="907c8310-2e6e-4ce4-8a52-0c08a77ff0c8" containerName="revoke-keys"
Nov 02 22:05:03.457460 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.457437    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:03.505346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.505312    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"]
Nov 02 22:05:03.552714 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.552682    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:05:03.552714 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.552705    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:05:03.552888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.552687    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:05:03.560832 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.560809    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:05:03.612890 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.612865    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45e7e934-q5749\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:03.613013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.612928    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6j9\" (UniqueName: \"kubernetes.io/projected/4ea41520-417b-475d-8c6e-c8343694fa58-kube-api-access-8m6j9\") pod \"maas-api-revoke-sub-45e7e934-q5749\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:03.613099 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.613080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45e7e934-q5749\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:03.719467 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.719370    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45e7e934-q5749\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:03.719762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.719737    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6j9\" (UniqueName: \"kubernetes.io/projected/4ea41520-417b-475d-8c6e-c8343694fa58-kube-api-access-8m6j9\") pod \"maas-api-revoke-sub-45e7e934-q5749\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:03.725638 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.720837    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45e7e934-q5749\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:03.725918 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.725702    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45e7e934-q5749\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:03.730597 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.730571    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45e7e934-q5749\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:03.777474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:03.777444    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6j9\" (UniqueName: \"kubernetes.io/projected/4ea41520-417b-475d-8c6e-c8343694fa58-kube-api-access-8m6j9\") pod \"maas-api-revoke-sub-45e7e934-q5749\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:04.072816 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:04.072783    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:04.445764 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:05:04.445735    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea41520_417b_475d_8c6e_c8343694fa58.slice/crio-b54f18951d9156762a9cacc7084c10df2b0e3054ac29496a766e1a95886b84db WatchSource:0}: Error finding container b54f18951d9156762a9cacc7084c10df2b0e3054ac29496a766e1a95886b84db: Status 404 returned error can't find the container with id b54f18951d9156762a9cacc7084c10df2b0e3054ac29496a766e1a95886b84db
Nov 02 22:05:04.449400 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:04.449375    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"]
Nov 02 22:05:04.737190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:04.737151    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749" event={"ID":"4ea41520-417b-475d-8c6e-c8343694fa58","Type":"ContainerStarted","Data":"280bfdedbced75a46e50e51e952bc226af4e86f536c43119e4474b187e90402d"}
Nov 02 22:05:04.737517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:04.737214    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749" event={"ID":"4ea41520-417b-475d-8c6e-c8343694fa58","Type":"ContainerStarted","Data":"b54f18951d9156762a9cacc7084c10df2b0e3054ac29496a766e1a95886b84db"}
Nov 02 22:05:04.853313 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:04.853265    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749" podStartSLOduration=1.85325119 podStartE2EDuration="1.85325119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:04.851746316 +0000 UTC m=+3392.115282633" watchObservedRunningTime="2026-11-02 22:05:04.85325119 +0000 UTC m=+3392.116787508"
Nov 02 22:05:05.744729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:05.744632    4528 generic.go:356] "Generic (PLEG): container finished" podID="4ea41520-417b-475d-8c6e-c8343694fa58" containerID="280bfdedbced75a46e50e51e952bc226af4e86f536c43119e4474b187e90402d" exitCode=0
Nov 02 22:05:05.744729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:05.744709    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749" event={"ID":"4ea41520-417b-475d-8c6e-c8343694fa58","Type":"ContainerDied","Data":"280bfdedbced75a46e50e51e952bc226af4e86f536c43119e4474b187e90402d"}
Nov 02 22:05:06.914132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:06.914094    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:06.958840 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:06.958811    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-openshift-ca\") pod \"4ea41520-417b-475d-8c6e-c8343694fa58\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") "
Nov 02 22:05:06.959011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:06.958860    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-xks-ca\") pod \"4ea41520-417b-475d-8c6e-c8343694fa58\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") "
Nov 02 22:05:06.959011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:06.958993    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea41520-417b-475d-8c6e-c8343694fa58-kube-api-access-8m6j9\" (UniqueName: \"kubernetes.io/projected/4ea41520-417b-475d-8c6e-c8343694fa58-kube-api-access-8m6j9\") pod \"4ea41520-417b-475d-8c6e-c8343694fa58\" (UID: \"4ea41520-417b-475d-8c6e-c8343694fa58\") "
Nov 02 22:05:06.959244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:06.959222    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-openshift-ca" pod "4ea41520-417b-475d-8c6e-c8343694fa58" (UID: "4ea41520-417b-475d-8c6e-c8343694fa58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:05:06.959448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:06.959432    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:06.960926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:06.960898    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-xks-ca" pod "4ea41520-417b-475d-8c6e-c8343694fa58" (UID: "4ea41520-417b-475d-8c6e-c8343694fa58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:05:06.961047 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:06.961022    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea41520-417b-475d-8c6e-c8343694fa58-kube-api-access-8m6j9" pod "4ea41520-417b-475d-8c6e-c8343694fa58" (UID: "4ea41520-417b-475d-8c6e-c8343694fa58"). InnerVolumeSpecName "kube-api-access-8m6j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:05:07.060866 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:07.060837    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ea41520-417b-475d-8c6e-c8343694fa58-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:07.061040 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:07.060869    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6j9\" (UniqueName: \"kubernetes.io/projected/4ea41520-417b-475d-8c6e-c8343694fa58-kube-api-access-8m6j9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:07.769133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:07.769096    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"
Nov 02 22:05:07.769285 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:07.769130    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749" event={"ID":"4ea41520-417b-475d-8c6e-c8343694fa58","Type":"ContainerDied","Data":"b54f18951d9156762a9cacc7084c10df2b0e3054ac29496a766e1a95886b84db"}
Nov 02 22:05:07.769285 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:07.769195    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b54f18951d9156762a9cacc7084c10df2b0e3054ac29496a766e1a95886b84db"
Nov 02 22:05:08.771088 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.771056    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"]
Nov 02 22:05:08.771907 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.771885    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4ea41520-417b-475d-8c6e-c8343694fa58" containerName="revoke-keys"
Nov 02 22:05:08.771961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.771911    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea41520-417b-475d-8c6e-c8343694fa58" containerName="revoke-keys"
Nov 02 22:05:08.772054 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.772043    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea41520-417b-475d-8c6e-c8343694fa58" containerName="revoke-keys"
Nov 02 22:05:08.775852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.775832    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.776798 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.776778    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-podinfo\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.776897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.776831    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56d731-5280-4acc-bcda-a5eba4540f36-istiod-ca-cert\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.776956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.776936    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-token\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.777011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.776996    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-data\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.777100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.777083    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfg7w\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-kube-api-access-qfg7w\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.777166 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.777145    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-socket\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.777203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.777176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-certs\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.777282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.777271    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-envoy\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.777318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.777300    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-credential-socket\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.831399 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.831367    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"]
Nov 02 22:05:08.875668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.875641    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9981d2-openshift-default-dockercfg-9q4mx\""
Nov 02 22:05:08.878313 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.878285    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-envoy\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.878385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.878340    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-credential-socket\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.878430 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.878417    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-podinfo\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.878474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.878463    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56d731-5280-4acc-bcda-a5eba4540f36-istiod-ca-cert\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.878522 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.878502    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-token\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.878570 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.878552    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-data\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.878626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.878612    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfg7w\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-kube-api-access-qfg7w\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.878806 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.878780    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-credential-socket\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.878987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.878957    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-data\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.879100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.879025    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-socket\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.879100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.879084    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-certs\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.879280 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.879139    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56d731-5280-4acc-bcda-a5eba4540f36-istiod-ca-cert\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.879350 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.879334    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-socket\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.879421 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.879399    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-certs\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.881023 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.881006    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-envoy\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.881204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.881186    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-podinfo\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.942624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.942556    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-token\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:08.943488 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:08.943436    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfg7w\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-kube-api-access-qfg7w\") pod \"e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") " pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:09.089771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:09.089680    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:09.530875 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:05:09.530836    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d56d731_5280_4acc_bcda_a5eba4540f36.slice/crio-28b07a821d3c7c1d4d3dc48b091c6af63cfc551c67ac0b7cf5e9ac5678a1b3ea WatchSource:0}: Error finding container 28b07a821d3c7c1d4d3dc48b091c6af63cfc551c67ac0b7cf5e9ac5678a1b3ea: Status 404 returned error can't find the container with id 28b07a821d3c7c1d4d3dc48b091c6af63cfc551c67ac0b7cf5e9ac5678a1b3ea
Nov 02 22:05:09.533541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:09.533507    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:05:09.533661 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:09.533576    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:05:09.533661 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:09.533605    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:05:09.533983 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:09.533958    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"]
Nov 02 22:05:09.784338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:09.784239    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" event={"ID":"0d56d731-5280-4acc-bcda-a5eba4540f36","Type":"ContainerStarted","Data":"3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f"}
Nov 02 22:05:09.784338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:09.784295    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" event={"ID":"0d56d731-5280-4acc-bcda-a5eba4540f36","Type":"ContainerStarted","Data":"28b07a821d3c7c1d4d3dc48b091c6af63cfc551c67ac0b7cf5e9ac5678a1b3ea"}
Nov 02 22:05:09.901479 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:09.901423    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" podStartSLOduration=1.901408656 podStartE2EDuration="1.901408656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:09.90006442 +0000 UTC m=+3397.163600739" watchObservedRunningTime="2026-11-02 22:05:09.901408656 +0000 UTC m=+3397.164944973"
Nov 02 22:05:10.090004 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:10.089913    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:10.095697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:10.095669    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:10.793004 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:10.792977    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:10.793990 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:10.793969    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:05:14.140960 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:05:14.140891    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:05:16.258346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.258307    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"]
Nov 02 22:05:16.262509 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.262488    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:16.316480 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.316446    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"]
Nov 02 22:05:16.370780 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.370714    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9981d2\""
Nov 02 22:05:16.373385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.373355    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9981d2-dockercfg-2phmp\""
Nov 02 22:05:16.426976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.426941    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"]
Nov 02 22:05:16.432179 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.432145    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:16.453169 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.453133    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pg7c\" (UniqueName: \"kubernetes.io/projected/7c6d22ef-d7bb-42dd-acba-8a5479241c98-kube-api-access-8pg7c\") pod \"payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp\" (UID: \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:16.453364 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.453345    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6d22ef-d7bb-42dd-acba-8a5479241c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp\" (UID: \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:16.482936 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.482904    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"]
Nov 02 22:05:16.555022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.554986    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cc8b19-c952-46f2-a316-310cc3254e87-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv\" (UID: \"c0cc8b19-c952-46f2-a316-310cc3254e87\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:16.555245 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.555140    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pg7c\" (UniqueName: \"kubernetes.io/projected/7c6d22ef-d7bb-42dd-acba-8a5479241c98-kube-api-access-8pg7c\") pod \"payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp\" (UID: \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:16.555245 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.555186    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzx8p\" (UniqueName: \"kubernetes.io/projected/c0cc8b19-c952-46f2-a316-310cc3254e87-kube-api-access-rzx8p\") pod \"payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv\" (UID: \"c0cc8b19-c952-46f2-a316-310cc3254e87\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:16.555245 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.555223    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6d22ef-d7bb-42dd-acba-8a5479241c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp\" (UID: \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:16.556154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.556134    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6d22ef-d7bb-42dd-acba-8a5479241c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp\" (UID: \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:16.611691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.611660    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pg7c\" (UniqueName: \"kubernetes.io/projected/7c6d22ef-d7bb-42dd-acba-8a5479241c98-kube-api-access-8pg7c\") pod \"payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp\" (UID: \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:16.656203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.656167    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzx8p\" (UniqueName: \"kubernetes.io/projected/c0cc8b19-c952-46f2-a316-310cc3254e87-kube-api-access-rzx8p\") pod \"payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv\" (UID: \"c0cc8b19-c952-46f2-a316-310cc3254e87\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:16.656382 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.656329    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cc8b19-c952-46f2-a316-310cc3254e87-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv\" (UID: \"c0cc8b19-c952-46f2-a316-310cc3254e87\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:16.657039 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.657017    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cc8b19-c952-46f2-a316-310cc3254e87-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv\" (UID: \"c0cc8b19-c952-46f2-a316-310cc3254e87\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:16.713490 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.713455    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzx8p\" (UniqueName: \"kubernetes.io/projected/c0cc8b19-c952-46f2-a316-310cc3254e87-kube-api-access-rzx8p\") pod \"payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv\" (UID: \"c0cc8b19-c952-46f2-a316-310cc3254e87\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:16.748304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.748268    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:16.877295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:16.877259    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:17.122651 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:05:17.119228    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cc8b19_c952_46f2_a316_310cc3254e87.slice/crio-bf3a9e206907811b0cb157c3afc6648464543430cfd9c596f3b3acdf1dc49580 WatchSource:0}: Error finding container bf3a9e206907811b0cb157c3afc6648464543430cfd9c596f3b3acdf1dc49580: Status 404 returned error can't find the container with id bf3a9e206907811b0cb157c3afc6648464543430cfd9c596f3b3acdf1dc49580
Nov 02 22:05:17.122859 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:17.122757    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"]
Nov 02 22:05:17.245336 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:05:17.245290    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6d22ef_d7bb_42dd_acba_8a5479241c98.slice/crio-aba1ac4286ad60eacb31d05671008e434c243b7fdc23acb628d1a5b371fee918 WatchSource:0}: Error finding container aba1ac4286ad60eacb31d05671008e434c243b7fdc23acb628d1a5b371fee918: Status 404 returned error can't find the container with id aba1ac4286ad60eacb31d05671008e434c243b7fdc23acb628d1a5b371fee918
Nov 02 22:05:17.248499 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:17.248471    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"]
Nov 02 22:05:17.842161 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:17.842124    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv" event={"ID":"c0cc8b19-c952-46f2-a316-310cc3254e87","Type":"ContainerStarted","Data":"05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef"}
Nov 02 22:05:17.842627 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:17.842178    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv" event={"ID":"c0cc8b19-c952-46f2-a316-310cc3254e87","Type":"ContainerStarted","Data":"bf3a9e206907811b0cb157c3afc6648464543430cfd9c596f3b3acdf1dc49580"}
Nov 02 22:05:17.842627 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:17.842211    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:17.843894 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:17.843870    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp" event={"ID":"7c6d22ef-d7bb-42dd-acba-8a5479241c98","Type":"ContainerStarted","Data":"56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68"}
Nov 02 22:05:17.844014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:17.843909    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp" event={"ID":"7c6d22ef-d7bb-42dd-acba-8a5479241c98","Type":"ContainerStarted","Data":"aba1ac4286ad60eacb31d05671008e434c243b7fdc23acb628d1a5b371fee918"}
Nov 02 22:05:17.844014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:17.843942    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:17.957469 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:17.957418    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv" podStartSLOduration=1.957404313 podStartE2EDuration="1.957404313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:17.955417914 +0000 UTC m=+3405.218954245" watchObservedRunningTime="2026-11-02 22:05:17.957404313 +0000 UTC m=+3405.220940631"
Nov 02 22:05:18.087268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:18.087210    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp" podStartSLOduration=2.087190229 podStartE2EDuration="2.087190229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:18.086550479 +0000 UTC m=+3405.350086801" watchObservedRunningTime="2026-11-02 22:05:18.087190229 +0000 UTC m=+3405.350726548"
Nov 02 22:05:28.853690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:28.853656    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:05:28.854187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:28.853991    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:05:29.467976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.467943    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"]
Nov 02 22:05:29.472420 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.472401    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:29.523671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.523636    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"]
Nov 02 22:05:29.580407 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.571309    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-5ddbd0-dockercfg-67mkl\""
Nov 02 22:05:29.580928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.580889    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-5ddbd0\""
Nov 02 22:05:29.592865 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.592830    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b59ct\" (UniqueName: \"kubernetes.io/projected/6cf37c52-d094-41b5-bfc7-48fab52419ee-kube-api-access-b59ct\") pod \"payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww\" (UID: \"6cf37c52-d094-41b5-bfc7-48fab52419ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:29.595783 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.595639    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf37c52-d094-41b5-bfc7-48fab52419ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww\" (UID: \"6cf37c52-d094-41b5-bfc7-48fab52419ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:29.648133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.644537    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"]
Nov 02 22:05:29.651051 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.649200    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:29.683997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.683965    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"]
Nov 02 22:05:29.697895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.697856    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b59ct\" (UniqueName: \"kubernetes.io/projected/6cf37c52-d094-41b5-bfc7-48fab52419ee-kube-api-access-b59ct\") pod \"payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww\" (UID: \"6cf37c52-d094-41b5-bfc7-48fab52419ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:29.698091 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.697973    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19974f7-0c80-4573-968d-350048867d30-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52\" (UID: \"d19974f7-0c80-4573-968d-350048867d30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:29.698091 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.698055    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf37c52-d094-41b5-bfc7-48fab52419ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww\" (UID: \"6cf37c52-d094-41b5-bfc7-48fab52419ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:29.698277 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.698096    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dx6x\" (UniqueName: \"kubernetes.io/projected/d19974f7-0c80-4573-968d-350048867d30-kube-api-access-2dx6x\") pod \"payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52\" (UID: \"d19974f7-0c80-4573-968d-350048867d30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:29.699236 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.699204    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf37c52-d094-41b5-bfc7-48fab52419ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww\" (UID: \"6cf37c52-d094-41b5-bfc7-48fab52419ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:29.754722 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.754650    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b59ct\" (UniqueName: \"kubernetes.io/projected/6cf37c52-d094-41b5-bfc7-48fab52419ee-kube-api-access-b59ct\") pod \"payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww\" (UID: \"6cf37c52-d094-41b5-bfc7-48fab52419ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:29.786925 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.786889    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:29.800661 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.800633    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19974f7-0c80-4573-968d-350048867d30-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52\" (UID: \"d19974f7-0c80-4573-968d-350048867d30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:29.800822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.800711    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dx6x\" (UniqueName: \"kubernetes.io/projected/d19974f7-0c80-4573-968d-350048867d30-kube-api-access-2dx6x\") pod \"payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52\" (UID: \"d19974f7-0c80-4573-968d-350048867d30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:29.801370 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.801347    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19974f7-0c80-4573-968d-350048867d30-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52\" (UID: \"d19974f7-0c80-4573-968d-350048867d30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:29.857916 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.857892    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dx6x\" (UniqueName: \"kubernetes.io/projected/d19974f7-0c80-4573-968d-350048867d30-kube-api-access-2dx6x\") pod \"payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52\" (UID: \"d19974f7-0c80-4573-968d-350048867d30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:29.966343 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:29.966308    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:30.162273 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:05:30.162237    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf37c52_d094_41b5_bfc7_48fab52419ee.slice/crio-c691f5b04c816a8c88978c1708a101d14db5e4cd5728c2863a1a52c8491155c1 WatchSource:0}: Error finding container c691f5b04c816a8c88978c1708a101d14db5e4cd5728c2863a1a52c8491155c1: Status 404 returned error can't find the container with id c691f5b04c816a8c88978c1708a101d14db5e4cd5728c2863a1a52c8491155c1
Nov 02 22:05:30.165236 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:30.165210    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"]
Nov 02 22:05:30.339865 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:05:30.339831    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19974f7_0c80_4573_968d_350048867d30.slice/crio-5d87836b44dd0f5ae1e3dc1a30dae7682d2e95ed326c1f266cab6e3fbc34d930 WatchSource:0}: Error finding container 5d87836b44dd0f5ae1e3dc1a30dae7682d2e95ed326c1f266cab6e3fbc34d930: Status 404 returned error can't find the container with id 5d87836b44dd0f5ae1e3dc1a30dae7682d2e95ed326c1f266cab6e3fbc34d930
Nov 02 22:05:30.342002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:30.341978    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"]
Nov 02 22:05:30.928797 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:30.928753    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52" event={"ID":"d19974f7-0c80-4573-968d-350048867d30","Type":"ContainerStarted","Data":"022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4"}
Nov 02 22:05:30.929301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:30.928832    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:30.929301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:30.928858    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52" event={"ID":"d19974f7-0c80-4573-968d-350048867d30","Type":"ContainerStarted","Data":"5d87836b44dd0f5ae1e3dc1a30dae7682d2e95ed326c1f266cab6e3fbc34d930"}
Nov 02 22:05:30.930560 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:30.930531    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww" event={"ID":"6cf37c52-d094-41b5-bfc7-48fab52419ee","Type":"ContainerStarted","Data":"58fb79312276d93c57c33acac94290a234c860ec6b6b2158f8ab21c46b9179e0"}
Nov 02 22:05:30.930692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:30.930572    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww" event={"ID":"6cf37c52-d094-41b5-bfc7-48fab52419ee","Type":"ContainerStarted","Data":"c691f5b04c816a8c88978c1708a101d14db5e4cd5728c2863a1a52c8491155c1"}
Nov 02 22:05:30.930692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:30.930609    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:31.042420 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:31.042366    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52" podStartSLOduration=2.042352452 podStartE2EDuration="2.042352452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:31.041765863 +0000 UTC m=+3418.305302180" watchObservedRunningTime="2026-11-02 22:05:31.042352452 +0000 UTC m=+3418.305888796"
Nov 02 22:05:31.182436 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:31.182290    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww" podStartSLOduration=2.182263079 podStartE2EDuration="2.182263079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:31.163041824 +0000 UTC m=+3418.426578156" watchObservedRunningTime="2026-11-02 22:05:31.182263079 +0000 UTC m=+3418.445799414"
Nov 02 22:05:31.341386 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:31.341350    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"]
Nov 02 22:05:40.761670 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.761638    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"]
Nov 02 22:05:40.765919 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.765900    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.861145 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.860402    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1538c4-openshift-default-dockercfg-fv5q5\""
Nov 02 22:05:40.870134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.868015    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-certs\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.870134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.868073    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505f97a2-29fe-4d06-a733-a182a3b87f03-istiod-ca-cert\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.870134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.868127    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-socket\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.870134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.868155    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-token\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.870134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.868243    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-podinfo\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.870134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.868346    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdb57\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-kube-api-access-cdb57\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.870134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.868478    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-envoy\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.870134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.868600    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-credential-socket\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.870134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.868623    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-data\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.886131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.884895    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"]
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.970411    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-envoy\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.970490    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-credential-socket\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.970509    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-data\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.970543    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-certs\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.970562    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505f97a2-29fe-4d06-a733-a182a3b87f03-istiod-ca-cert\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.970579    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-socket\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.970597    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-token\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.970631    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-podinfo\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.970675    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdb57\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-kube-api-access-cdb57\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.971630    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-data\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.971882    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505f97a2-29fe-4d06-a733-a182a3b87f03-istiod-ca-cert\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.971882    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-credential-socket\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.972283    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-certs\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.974055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.972998    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-socket\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.981926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.978868    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-podinfo\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:40.981926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:40.981601    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-envoy\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:41.028397 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:41.028314    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdb57\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-kube-api-access-cdb57\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:41.028541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:41.028478    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-token\") pod \"e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") " pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:41.079348 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:41.079308    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:41.514675 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:05:41.514637    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505f97a2_29fe_4d06_a733_a182a3b87f03.slice/crio-887ef1053e88b72fac7ee36d9ae00416109847ecb4f361c3a8d3ed53e9d838dc WatchSource:0}: Error finding container 887ef1053e88b72fac7ee36d9ae00416109847ecb4f361c3a8d3ed53e9d838dc: Status 404 returned error can't find the container with id 887ef1053e88b72fac7ee36d9ae00416109847ecb4f361c3a8d3ed53e9d838dc
Nov 02 22:05:41.517467 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:41.517435    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:05:41.517563 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:41.517504    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:05:41.517563 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:41.517533    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:05:41.517917 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:41.517892    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"]
Nov 02 22:05:41.940954 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:41.940851    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:05:41.940954 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:41.940932    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:05:42.002976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:42.002936    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" event={"ID":"505f97a2-29fe-4d06-a733-a182a3b87f03","Type":"ContainerStarted","Data":"a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a"}
Nov 02 22:05:42.003208 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:42.003003    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" event={"ID":"505f97a2-29fe-4d06-a733-a182a3b87f03","Type":"ContainerStarted","Data":"887ef1053e88b72fac7ee36d9ae00416109847ecb4f361c3a8d3ed53e9d838dc"}
Nov 02 22:05:42.080122 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:42.080064    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:42.086297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:42.086264    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:42.287062 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:42.287007    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" podStartSLOduration=2.286990355 podStartE2EDuration="2.286990355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:42.28286491 +0000 UTC m=+3429.546401239" watchObservedRunningTime="2026-11-02 22:05:42.286990355 +0000 UTC m=+3429.550526673"
Nov 02 22:05:43.009872 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:43.009841    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:43.010753 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:43.010735    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:05:44.114642 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:05:44.114600    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:05:46.362868 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.362821    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47" podUID="b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" containerName="main" containerID="cri-o://648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17" gracePeriod=45
Nov 02 22:05:46.630266 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.630206    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 22:05:46.670788 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.670745    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-model-cache\") pod \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") "
Nov 02 22:05:46.670962 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.670812    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tls-certs\") pod \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") "
Nov 02 22:05:46.670962 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.670841    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-home\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-home\") pod \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") "
Nov 02 22:05:46.670962 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.670885    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-kube-api-access-bxqkw\" (UniqueName: \"kubernetes.io/projected/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-kube-api-access-bxqkw\") pod \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") "
Nov 02 22:05:46.670962 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.670958    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tmp-dir\") pod \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") "
Nov 02 22:05:46.671257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.670992    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-dshm\") pod \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\" (UID: \"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32\") "
Nov 02 22:05:46.671257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.671055    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-model-cache" pod "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" (UID: "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:05:46.671257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.671084    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-home" pod "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" (UID: "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:05:46.671257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.671243    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tmp-dir" pod "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" (UID: "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:05:46.671480 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.671461    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:46.671526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.671485    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:46.671526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.671500    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:46.673278 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.673252    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tls-certs" pod "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" (UID: "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:05:46.673278 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.673257    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-dshm" pod "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" (UID: "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:05:46.673434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.673308    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-kube-api-access-bxqkw" pod "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" (UID: "b0625b31-c3c2-4d5a-9192-4ab8c3d67a32"). InnerVolumeSpecName "kube-api-access-bxqkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:05:46.772698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.772644    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:46.772698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.772687    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:46.772698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:46.772701    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxqkw\" (UniqueName: \"kubernetes.io/projected/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32-kube-api-access-bxqkw\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:47.038028 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:47.037991    4528 generic.go:356] "Generic (PLEG): container finished" podID="b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" containerID="648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17" exitCode=0
Nov 02 22:05:47.038269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:47.038043    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"
Nov 02 22:05:47.038269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:47.038073    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47" event={"ID":"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32","Type":"ContainerDied","Data":"648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17"}
Nov 02 22:05:47.038269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:47.038129    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47" event={"ID":"b0625b31-c3c2-4d5a-9192-4ab8c3d67a32","Type":"ContainerDied","Data":"70f56a26086ea789bbe146a935a59db2f9ac54fc74eccaa8df508c54347b21aa"}
Nov 02 22:05:47.038269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:47.038147    4528 scope.go:122] "RemoveContainer" containerID="648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17"
Nov 02 22:05:47.048445 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:47.048422    4528 scope.go:122] "RemoveContainer" containerID="648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17"
Nov 02 22:05:47.048738 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:05:47.048717    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17\": container with ID starting with 648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17 not found: ID does not exist" containerID="648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17"
Nov 02 22:05:47.048807 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:47.048746    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17"} err="failed to get container status \"648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17\": rpc error: code = NotFound desc = could not find container \"648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17\": container with ID starting with 648949cfe58ed661ade794c0284ec14ad6b2e400c8d23ca0bda37f15b1a9fa17 not found: ID does not exist"
Nov 02 22:05:47.203667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:47.203548    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"]
Nov 02 22:05:47.208901 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:47.208872    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/auth-test-model-7cde84-kserve-65449b46d5-dzb47"]
Nov 02 22:05:48.914226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:48.914187    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" path="/var/lib/kubelet/pods/b0625b31-c3c2-4d5a-9192-4ab8c3d67a32/volumes"
Nov 02 22:05:48.951865 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:48.951833    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"]
Nov 02 22:05:48.952510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:48.952494    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" containerName="main"
Nov 02 22:05:48.952564 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:48.952511    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" containerName="main"
Nov 02 22:05:48.952631 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:48.952622    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0625b31-c3c2-4d5a-9192-4ab8c3d67a32" containerName="main"
Nov 02 22:05:48.957166 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:48.957146    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:05:48.999280 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:48.999242    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7ced3-faf8-4292-ae0d-7109ae65a57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8\" (UID: \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:05:48.999462 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:48.999320    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkf2n\" (UniqueName: \"kubernetes.io/projected/12a7ced3-faf8-4292-ae0d-7109ae65a57a-kube-api-access-dkf2n\") pod \"payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8\" (UID: \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:05:49.009545 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.009503    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"]
Nov 02 22:05:49.055749 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.055718    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-1538c4\""
Nov 02 22:05:49.056167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.056142    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-1538c4-dockercfg-kqqsz\""
Nov 02 22:05:49.128156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.117514    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7ced3-faf8-4292-ae0d-7109ae65a57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8\" (UID: \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:05:49.129726 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.129461    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7ced3-faf8-4292-ae0d-7109ae65a57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8\" (UID: \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:05:49.136878 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.129741    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkf2n\" (UniqueName: \"kubernetes.io/projected/12a7ced3-faf8-4292-ae0d-7109ae65a57a-kube-api-access-dkf2n\") pod \"payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8\" (UID: \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:05:49.142915 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.142879    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"]
Nov 02 22:05:49.148541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.148502    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:05:49.170483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.170409    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"]
Nov 02 22:05:49.186430 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.186391    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkf2n\" (UniqueName: \"kubernetes.io/projected/12a7ced3-faf8-4292-ae0d-7109ae65a57a-kube-api-access-dkf2n\") pod \"payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8\" (UID: \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:05:49.235771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.235733    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f1f08-46db-4b28-934a-0e59117c6726-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7\" (UID: \"672f1f08-46db-4b28-934a-0e59117c6726\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:05:49.235771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.235779    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjrd\" (UniqueName: \"kubernetes.io/projected/672f1f08-46db-4b28-934a-0e59117c6726-kube-api-access-gmjrd\") pod \"payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7\" (UID: \"672f1f08-46db-4b28-934a-0e59117c6726\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:05:49.271718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.271686    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:05:49.343830 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.337255    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f1f08-46db-4b28-934a-0e59117c6726-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7\" (UID: \"672f1f08-46db-4b28-934a-0e59117c6726\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:05:49.343830 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.337319    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjrd\" (UniqueName: \"kubernetes.io/projected/672f1f08-46db-4b28-934a-0e59117c6726-kube-api-access-gmjrd\") pod \"payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7\" (UID: \"672f1f08-46db-4b28-934a-0e59117c6726\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:05:49.343830 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.338425    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f1f08-46db-4b28-934a-0e59117c6726-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7\" (UID: \"672f1f08-46db-4b28-934a-0e59117c6726\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:05:49.396099 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.396062    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjrd\" (UniqueName: \"kubernetes.io/projected/672f1f08-46db-4b28-934a-0e59117c6726-kube-api-access-gmjrd\") pod \"payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7\" (UID: \"672f1f08-46db-4b28-934a-0e59117c6726\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:05:49.469079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.468991    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:05:49.647544 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:05:49.647508    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a7ced3_faf8_4292_ae0d_7109ae65a57a.slice/crio-5cfaa08f107a58324cbe5643f252e52002dfaa60b6b14e275bad615145c3ca5f WatchSource:0}: Error finding container 5cfaa08f107a58324cbe5643f252e52002dfaa60b6b14e275bad615145c3ca5f: Status 404 returned error can't find the container with id 5cfaa08f107a58324cbe5643f252e52002dfaa60b6b14e275bad615145c3ca5f
Nov 02 22:05:49.650526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.650501    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"]
Nov 02 22:05:49.841504 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:05:49.841468    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672f1f08_46db_4b28_934a_0e59117c6726.slice/crio-71a52cc39054ed38fde7c1f74c014636cc5f0f98c384e6c62d627a454f86f52b WatchSource:0}: Error finding container 71a52cc39054ed38fde7c1f74c014636cc5f0f98c384e6c62d627a454f86f52b: Status 404 returned error can't find the container with id 71a52cc39054ed38fde7c1f74c014636cc5f0f98c384e6c62d627a454f86f52b
Nov 02 22:05:49.844208 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:49.844180    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"]
Nov 02 22:05:50.061619 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:50.061577    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7" event={"ID":"672f1f08-46db-4b28-934a-0e59117c6726","Type":"ContainerStarted","Data":"020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a"}
Nov 02 22:05:50.062078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:50.061652    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7" event={"ID":"672f1f08-46db-4b28-934a-0e59117c6726","Type":"ContainerStarted","Data":"71a52cc39054ed38fde7c1f74c014636cc5f0f98c384e6c62d627a454f86f52b"}
Nov 02 22:05:50.062078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:50.061815    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:05:50.063160 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:50.063138    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8" event={"ID":"12a7ced3-faf8-4292-ae0d-7109ae65a57a","Type":"ContainerStarted","Data":"de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438"}
Nov 02 22:05:50.063259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:50.063188    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8" event={"ID":"12a7ced3-faf8-4292-ae0d-7109ae65a57a","Type":"ContainerStarted","Data":"5cfaa08f107a58324cbe5643f252e52002dfaa60b6b14e275bad615145c3ca5f"}
Nov 02 22:05:50.063259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:50.063220    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:05:50.177922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:50.177865    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7" podStartSLOduration=1.177850079 podStartE2EDuration="1.177850079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:50.174266803 +0000 UTC m=+3437.437803121" watchObservedRunningTime="2026-11-02 22:05:50.177850079 +0000 UTC m=+3437.441386414"
Nov 02 22:05:50.306137 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:50.305854    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8" podStartSLOduration=2.305829807 podStartE2EDuration="2.305829807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:50.304189799 +0000 UTC m=+3437.567726118" watchObservedRunningTime="2026-11-02 22:05:50.305829807 +0000 UTC m=+3437.569366128"
Nov 02 22:05:52.027587 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:52.027549    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"]
Nov 02 22:05:52.028052 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:52.027749    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" containerID="cri-o://f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5" gracePeriod=30
Nov 02 22:05:56.901794 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:56.901749    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:05:56.902222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:56.901811    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:05:57.125084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.125052    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:05:57.323503 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.323471    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 22:05:57.437230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437180    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-envoy\") pod \"ff97872b-bd19-40fe-9f23-cf44c308603c\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") "
Nov 02 22:05:57.437417 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437246    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-podinfo\") pod \"ff97872b-bd19-40fe-9f23-cf44c308603c\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") "
Nov 02 22:05:57.437417 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437378    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff97872b-bd19-40fe-9f23-cf44c308603c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97872b-bd19-40fe-9f23-cf44c308603c-istiod-ca-cert\") pod \"ff97872b-bd19-40fe-9f23-cf44c308603c\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") "
Nov 02 22:05:57.437540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437419    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-certs\") pod \"ff97872b-bd19-40fe-9f23-cf44c308603c\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") "
Nov 02 22:05:57.437540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437460    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-data\") pod \"ff97872b-bd19-40fe-9f23-cf44c308603c\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") "
Nov 02 22:05:57.437540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437510    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-socket\") pod \"ff97872b-bd19-40fe-9f23-cf44c308603c\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") "
Nov 02 22:05:57.437715 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437538    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-token\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-token\") pod \"ff97872b-bd19-40fe-9f23-cf44c308603c\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") "
Nov 02 22:05:57.437715 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437619    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-kube-api-access-pj7hf\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-kube-api-access-pj7hf\") pod \"ff97872b-bd19-40fe-9f23-cf44c308603c\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") "
Nov 02 22:05:57.437833 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437770    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-credential-socket\") pod \"ff97872b-bd19-40fe-9f23-cf44c308603c\" (UID: \"ff97872b-bd19-40fe-9f23-cf44c308603c\") "
Nov 02 22:05:57.437833 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437810    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-data" pod "ff97872b-bd19-40fe-9f23-cf44c308603c" (UID: "ff97872b-bd19-40fe-9f23-cf44c308603c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:05:57.437952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.437833    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-socket" pod "ff97872b-bd19-40fe-9f23-cf44c308603c" (UID: "ff97872b-bd19-40fe-9f23-cf44c308603c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:05:57.438404 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.438305    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-certs" pod "ff97872b-bd19-40fe-9f23-cf44c308603c" (UID: "ff97872b-bd19-40fe-9f23-cf44c308603c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:05:57.438404 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.438323    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:57.438404 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.438362    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:57.438705 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.438541    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-credential-socket" pod "ff97872b-bd19-40fe-9f23-cf44c308603c" (UID: "ff97872b-bd19-40fe-9f23-cf44c308603c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:05:57.439150 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.439093    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff97872b-bd19-40fe-9f23-cf44c308603c-istiod-ca-cert" pod "ff97872b-bd19-40fe-9f23-cf44c308603c" (UID: "ff97872b-bd19-40fe-9f23-cf44c308603c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:05:57.440044 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.440012    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-podinfo" pod "ff97872b-bd19-40fe-9f23-cf44c308603c" (UID: "ff97872b-bd19-40fe-9f23-cf44c308603c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:05:57.440190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.440171    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-envoy" pod "ff97872b-bd19-40fe-9f23-cf44c308603c" (UID: "ff97872b-bd19-40fe-9f23-cf44c308603c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:05:57.440259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.440232    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-token" pod "ff97872b-bd19-40fe-9f23-cf44c308603c" (UID: "ff97872b-bd19-40fe-9f23-cf44c308603c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:05:57.440367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.440349    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-kube-api-access-pj7hf" pod "ff97872b-bd19-40fe-9f23-cf44c308603c" (UID: "ff97872b-bd19-40fe-9f23-cf44c308603c"). InnerVolumeSpecName "kube-api-access-pj7hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:05:57.539754 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.539715    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97872b-bd19-40fe-9f23-cf44c308603c-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:57.539754 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.539749    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:57.539754 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.539761    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:57.540020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.539773    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj7hf\" (UniqueName: \"kubernetes.io/projected/ff97872b-bd19-40fe-9f23-cf44c308603c-kube-api-access-pj7hf\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:57.540020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.539787    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:57.540020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.539799    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:57.540020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:57.539809    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97872b-bd19-40fe-9f23-cf44c308603c-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:05:58.117712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.117674    4528 generic.go:356] "Generic (PLEG): container finished" podID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerID="f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5" exitCode=0
Nov 02 22:05:58.118174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.117744    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"
Nov 02 22:05:58.118174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.117759    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" event={"ID":"ff97872b-bd19-40fe-9f23-cf44c308603c","Type":"ContainerDied","Data":"f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5"}
Nov 02 22:05:58.118174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.117824    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2" event={"ID":"ff97872b-bd19-40fe-9f23-cf44c308603c","Type":"ContainerDied","Data":"09e481108d97f21f13782b0ffb4952918dd938c4b20bedab0def47a3dd14df6e"}
Nov 02 22:05:58.118174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.117847    4528 scope.go:122] "RemoveContainer" containerID="f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5"
Nov 02 22:05:58.129304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.129284    4528 scope.go:122] "RemoveContainer" containerID="f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5"
Nov 02 22:05:58.129572 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:05:58.129551    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5\": container with ID starting with f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5 not found: ID does not exist" containerID="f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5"
Nov 02 22:05:58.129722 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.129580    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5"} err="failed to get container status \"f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5\": rpc error: code = NotFound desc = could not find container \"f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5\": container with ID starting with f21fa5948bf6cbb4321e62043e337897c0f7ffb369506583fed7afbfabd727d5 not found: ID does not exist"
Nov 02 22:05:58.291977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.291947    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"]
Nov 02 22:05:58.348282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.348244    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-chtz2"]
Nov 02 22:05:58.913969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:05:58.913939    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" path="/var/lib/kubelet/pods/ff97872b-bd19-40fe-9f23-cf44c308603c/volumes"
Nov 02 22:06:01.073314 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:01.073284    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:06:01.073723 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:01.073354    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:06:14.113432 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:06:14.113395    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:06:42.341938 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.341899    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"]
Nov 02 22:06:42.342803 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.342780    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy"
Nov 02 22:06:42.342921 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.342803    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy"
Nov 02 22:06:42.343006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.342992    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff97872b-bd19-40fe-9f23-cf44c308603c" containerName="istio-proxy"
Nov 02 22:06:42.346532 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.346511    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:42.398252 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.398226    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"]
Nov 02 22:06:42.428795 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.428755    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5sbv\" (UniqueName: \"kubernetes.io/projected/cb5c2d52-04db-4583-baa8-9afd8a37dec1-kube-api-access-l5sbv\") pod \"payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf\" (UID: \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:42.428947 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.428912    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5c2d52-04db-4583-baa8-9afd8a37dec1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf\" (UID: \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:42.456909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.446414    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6ba06f-dockercfg-mqjjj\""
Nov 02 22:06:42.457654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.456831    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6ba06f\""
Nov 02 22:06:42.508033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.508001    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"]
Nov 02 22:06:42.520152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.520083    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:42.530505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.530267    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5sbv\" (UniqueName: \"kubernetes.io/projected/cb5c2d52-04db-4583-baa8-9afd8a37dec1-kube-api-access-l5sbv\") pod \"payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf\" (UID: \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:42.530505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.530407    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5c2d52-04db-4583-baa8-9afd8a37dec1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf\" (UID: \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:42.531222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.531196    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5c2d52-04db-4583-baa8-9afd8a37dec1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf\" (UID: \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:42.564459 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.564431    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"]
Nov 02 22:06:42.587400 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.587368    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5sbv\" (UniqueName: \"kubernetes.io/projected/cb5c2d52-04db-4583-baa8-9afd8a37dec1-kube-api-access-l5sbv\") pod \"payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf\" (UID: \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:42.631832 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.631753    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d\" (UID: \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:42.631978 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.631867    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqds6\" (UniqueName: \"kubernetes.io/projected/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-kube-api-access-dqds6\") pod \"payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d\" (UID: \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:42.660251 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.660229    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:42.733532 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.733501    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d\" (UID: \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:42.733667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.733603    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqds6\" (UniqueName: \"kubernetes.io/projected/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-kube-api-access-dqds6\") pod \"payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d\" (UID: \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:42.734398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.734379    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d\" (UID: \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:42.791362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.791337    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqds6\" (UniqueName: \"kubernetes.io/projected/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-kube-api-access-dqds6\") pod \"payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d\" (UID: \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:42.837455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:42.837432    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:43.034619 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:06:43.034590    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5c2d52_04db_4583_baa8_9afd8a37dec1.slice/crio-0646ff25e85bfe05cc2b1bd29a34f64b1b8c0f2bdcdcadb411b0ae98cb705f4e WatchSource:0}: Error finding container 0646ff25e85bfe05cc2b1bd29a34f64b1b8c0f2bdcdcadb411b0ae98cb705f4e: Status 404 returned error can't find the container with id 0646ff25e85bfe05cc2b1bd29a34f64b1b8c0f2bdcdcadb411b0ae98cb705f4e
Nov 02 22:06:43.037816 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.037790    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"]
Nov 02 22:06:43.209630 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:06:43.209595    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee7d18c_1ab7_40da_a3f3_9ad82080d8bc.slice/crio-abd9a5bf12034a784aa70cae8b2ce3463e8a567661a5b31ac95e4dc8c036c031 WatchSource:0}: Error finding container abd9a5bf12034a784aa70cae8b2ce3463e8a567661a5b31ac95e4dc8c036c031: Status 404 returned error can't find the container with id abd9a5bf12034a784aa70cae8b2ce3463e8a567661a5b31ac95e4dc8c036c031
Nov 02 22:06:43.211486 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.211465    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"]
Nov 02 22:06:43.418038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.418006    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d" event={"ID":"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc","Type":"ContainerStarted","Data":"0d818287dde90c2210e0405ea76a692dd2cb30339f70056d2fa7d0b4aeee1c14"}
Nov 02 22:06:43.418507 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.418069    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:43.418507 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.418092    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d" event={"ID":"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc","Type":"ContainerStarted","Data":"abd9a5bf12034a784aa70cae8b2ce3463e8a567661a5b31ac95e4dc8c036c031"}
Nov 02 22:06:43.419928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.419896    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf" event={"ID":"cb5c2d52-04db-4583-baa8-9afd8a37dec1","Type":"ContainerStarted","Data":"1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218"}
Nov 02 22:06:43.420056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.419944    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:43.420056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.419961    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf" event={"ID":"cb5c2d52-04db-4583-baa8-9afd8a37dec1","Type":"ContainerStarted","Data":"0646ff25e85bfe05cc2b1bd29a34f64b1b8c0f2bdcdcadb411b0ae98cb705f4e"}
Nov 02 22:06:43.532762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.532710    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d" podStartSLOduration=1.5326961099999998 podStartE2EDuration="1.53269611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:43.530787539 +0000 UTC m=+3490.794323857" watchObservedRunningTime="2026-11-02 22:06:43.53269611 +0000 UTC m=+3490.796232427"
Nov 02 22:06:43.652211 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:43.647868    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf" podStartSLOduration=1.647849307 podStartE2EDuration="1.647849307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:43.643438888 +0000 UTC m=+3490.906975206" watchObservedRunningTime="2026-11-02 22:06:43.647849307 +0000 UTC m=+3490.911385626"
Nov 02 22:06:44.114066 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:06:44.114031    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:06:54.429704 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:54.429677    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:06:54.430209 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:54.429745    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:06:55.209286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:06:55.209256    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:07:06.598531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:06.598497    4528 generic.go:356] "Generic (PLEG): container finished" podID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerID="3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f" exitCode=0
Nov 02 22:07:06.599009 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:06.598569    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" event={"ID":"0d56d731-5280-4acc-bcda-a5eba4540f36","Type":"ContainerDied","Data":"3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f"}
Nov 02 22:07:06.599009 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:06.599004    4528 scope.go:122] "RemoveContainer" containerID="3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f"
Nov 02 22:07:06.600694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:06.600669    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:07:06.600767 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:06.600742    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:07:06.600810 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:06.600773    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:07:07.632784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:07.632750    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" event={"ID":"0d56d731-5280-4acc-bcda-a5eba4540f36","Type":"ContainerStarted","Data":"96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d"}
Nov 02 22:07:08.090311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:08.090280    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:07:08.091644 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:08.091614    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Nov 02 22:07:08.091777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:08.091671    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Nov 02 22:07:09.090302 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:09.090269    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Nov 02 22:07:09.090793 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:09.090330    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Nov 02 22:07:10.090047 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:10.090011    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Nov 02 22:07:10.090296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:10.090071    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Nov 02 22:07:10.793450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:10.793413    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:07:11.090257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:11.090174    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Nov 02 22:07:11.090257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:11.090224    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Nov 02 22:07:12.090588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:12.090556    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Nov 02 22:07:12.090991 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:12.090608    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Nov 02 22:07:13.125706 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:13.125670    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:07:13.127574 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:13.127547    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:07:14.113639 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:07:14.113602    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:07:29.049120 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:29.049083    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:07:38.740020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.739984    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"]
Nov 02 22:07:38.744428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.744409    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:38.795336 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.795305    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"]
Nov 02 22:07:38.841905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.841865    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:07:38.841905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.841886    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:07:38.842124 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.841925    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:07:38.846875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.846861    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:07:38.920883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.920847    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1838cdc8-vhr6t\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:38.920883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.920883    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhml\" (UniqueName: \"kubernetes.io/projected/5158f93e-b3dc-438e-a87d-69798c0c5c50-kube-api-access-9zhml\") pod \"maas-api-revoke-sub-1838cdc8-vhr6t\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:38.921187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:38.921052    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1838cdc8-vhr6t\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:39.022148 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:39.022088    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1838cdc8-vhr6t\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:39.022339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:39.022157    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhml\" (UniqueName: \"kubernetes.io/projected/5158f93e-b3dc-438e-a87d-69798c0c5c50-kube-api-access-9zhml\") pod \"maas-api-revoke-sub-1838cdc8-vhr6t\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:39.022339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:39.022250    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1838cdc8-vhr6t\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:39.022930 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:39.022906    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1838cdc8-vhr6t\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:39.024592 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:39.024565    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1838cdc8-vhr6t\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:39.078875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:39.078838    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhml\" (UniqueName: \"kubernetes.io/projected/5158f93e-b3dc-438e-a87d-69798c0c5c50-kube-api-access-9zhml\") pod \"maas-api-revoke-sub-1838cdc8-vhr6t\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:39.358887 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:39.358804    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:39.729043 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:07:39.729001    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5158f93e_b3dc_438e_a87d_69798c0c5c50.slice/crio-3f67dc16a4d5eb7c3d97d718c58a86bd7b1e31d8ba31a0b3d59cc6580613b742 WatchSource:0}: Error finding container 3f67dc16a4d5eb7c3d97d718c58a86bd7b1e31d8ba31a0b3d59cc6580613b742: Status 404 returned error can't find the container with id 3f67dc16a4d5eb7c3d97d718c58a86bd7b1e31d8ba31a0b3d59cc6580613b742
Nov 02 22:07:39.732506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:39.732483    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"]
Nov 02 22:07:39.874940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:39.874905    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t" event={"ID":"5158f93e-b3dc-438e-a87d-69798c0c5c50","Type":"ContainerStarted","Data":"3f67dc16a4d5eb7c3d97d718c58a86bd7b1e31d8ba31a0b3d59cc6580613b742"}
Nov 02 22:07:40.883659 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:40.883571    4528 generic.go:356] "Generic (PLEG): container finished" podID="5158f93e-b3dc-438e-a87d-69798c0c5c50" containerID="b32036db68257273b00ab563eaa829fe67a9632129b95638151a53be3d345bd7" exitCode=0
Nov 02 22:07:40.884051 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:40.883664    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t" event={"ID":"5158f93e-b3dc-438e-a87d-69798c0c5c50","Type":"ContainerDied","Data":"b32036db68257273b00ab563eaa829fe67a9632129b95638151a53be3d345bd7"}
Nov 02 22:07:42.053531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.053504    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:42.065843 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.065821    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-openshift-ca\") pod \"5158f93e-b3dc-438e-a87d-69798c0c5c50\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") "
Nov 02 22:07:42.066167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.066145    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-openshift-ca" pod "5158f93e-b3dc-438e-a87d-69798c0c5c50" (UID: "5158f93e-b3dc-438e-a87d-69798c0c5c50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:07:42.066352 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.066333    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:07:42.167605 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.167528    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-xks-ca\") pod \"5158f93e-b3dc-438e-a87d-69798c0c5c50\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") "
Nov 02 22:07:42.167853 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.167605    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5158f93e-b3dc-438e-a87d-69798c0c5c50-kube-api-access-9zhml\" (UniqueName: \"kubernetes.io/projected/5158f93e-b3dc-438e-a87d-69798c0c5c50-kube-api-access-9zhml\") pod \"5158f93e-b3dc-438e-a87d-69798c0c5c50\" (UID: \"5158f93e-b3dc-438e-a87d-69798c0c5c50\") "
Nov 02 22:07:42.169855 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.169823    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-xks-ca" pod "5158f93e-b3dc-438e-a87d-69798c0c5c50" (UID: "5158f93e-b3dc-438e-a87d-69798c0c5c50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:07:42.170275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.170248    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5158f93e-b3dc-438e-a87d-69798c0c5c50-kube-api-access-9zhml" pod "5158f93e-b3dc-438e-a87d-69798c0c5c50" (UID: "5158f93e-b3dc-438e-a87d-69798c0c5c50"). InnerVolumeSpecName "kube-api-access-9zhml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:07:42.269613 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.269578    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5158f93e-b3dc-438e-a87d-69798c0c5c50-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:07:42.269613 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.269608    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zhml\" (UniqueName: \"kubernetes.io/projected/5158f93e-b3dc-438e-a87d-69798c0c5c50-kube-api-access-9zhml\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:07:42.897520 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.897481    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t" event={"ID":"5158f93e-b3dc-438e-a87d-69798c0c5c50","Type":"ContainerDied","Data":"3f67dc16a4d5eb7c3d97d718c58a86bd7b1e31d8ba31a0b3d59cc6580613b742"}
Nov 02 22:07:42.897520 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.897510    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"
Nov 02 22:07:42.897792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:42.897533    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f67dc16a4d5eb7c3d97d718c58a86bd7b1e31d8ba31a0b3d59cc6580613b742"
Nov 02 22:07:44.113596 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:07:44.113561    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:07:46.047559 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:46.047526    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"]
Nov 02 22:07:46.097808 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:46.097775    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64773fe5-nv7c6"]
Nov 02 22:07:46.914561 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:46.914529    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a898319-7458-4be4-898e-164e57dff464" path="/var/lib/kubelet/pods/3a898319-7458-4be4-898e-164e57dff464/volumes"
Nov 02 22:07:49.505350 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.505316    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"]
Nov 02 22:07:49.505966 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.505952    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5158f93e-b3dc-438e-a87d-69798c0c5c50" containerName="revoke-keys"
Nov 02 22:07:49.506022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.505967    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5158f93e-b3dc-438e-a87d-69798c0c5c50" containerName="revoke-keys"
Nov 02 22:07:49.506128 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.506117    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5158f93e-b3dc-438e-a87d-69798c0c5c50" containerName="revoke-keys"
Nov 02 22:07:49.509933 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.509918    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.558000 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.557965    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e28b1d-vpspv\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.558200 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.558100    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e28b1d-vpspv\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.558273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.558251    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5lz\" (UniqueName: \"kubernetes.io/projected/923a65af-abd7-4774-93f9-94597d6cfb9f-kube-api-access-gz5lz\") pod \"maas-api-revoke-sub-28e28b1d-vpspv\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.561282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.561258    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"]
Nov 02 22:07:49.607841 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.607810    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:07:49.608026 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.607846    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:07:49.614621 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.614601    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:07:49.614741 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.614654    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:07:49.659569 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.659535    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e28b1d-vpspv\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.659748 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.659602    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5lz\" (UniqueName: \"kubernetes.io/projected/923a65af-abd7-4774-93f9-94597d6cfb9f-kube-api-access-gz5lz\") pod \"maas-api-revoke-sub-28e28b1d-vpspv\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.659801 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.659745    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e28b1d-vpspv\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.660411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.660387    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e28b1d-vpspv\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.662038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.662017    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e28b1d-vpspv\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.717282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.717248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5lz\" (UniqueName: \"kubernetes.io/projected/923a65af-abd7-4774-93f9-94597d6cfb9f-kube-api-access-gz5lz\") pod \"maas-api-revoke-sub-28e28b1d-vpspv\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:49.823461 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:49.823434    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:50.196987 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:07:50.196936    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923a65af_abd7_4774_93f9_94597d6cfb9f.slice/crio-90a4fc79148b718f168fcae85e359d13b0723e1a0210567c348d86418bbf3557 WatchSource:0}: Error finding container 90a4fc79148b718f168fcae85e359d13b0723e1a0210567c348d86418bbf3557: Status 404 returned error can't find the container with id 90a4fc79148b718f168fcae85e359d13b0723e1a0210567c348d86418bbf3557
Nov 02 22:07:50.199932 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:50.199906    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"]
Nov 02 22:07:50.953812 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:50.953724    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv" event={"ID":"923a65af-abd7-4774-93f9-94597d6cfb9f","Type":"ContainerStarted","Data":"dfda084fcac97b09af08d47cfc92ef888c1f6d299f739bb803ec40de32203e32"}
Nov 02 22:07:50.953812 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:50.953786    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv" event={"ID":"923a65af-abd7-4774-93f9-94597d6cfb9f","Type":"ContainerStarted","Data":"90a4fc79148b718f168fcae85e359d13b0723e1a0210567c348d86418bbf3557"}
Nov 02 22:07:51.066632 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:51.066583    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv" podStartSLOduration=2.066569811 podStartE2EDuration="2.066569811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:51.066334644 +0000 UTC m=+3558.329870962" watchObservedRunningTime="2026-11-02 22:07:51.066569811 +0000 UTC m=+3558.330106128"
Nov 02 22:07:51.961796 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:51.961762    4528 generic.go:356] "Generic (PLEG): container finished" podID="923a65af-abd7-4774-93f9-94597d6cfb9f" containerID="dfda084fcac97b09af08d47cfc92ef888c1f6d299f739bb803ec40de32203e32" exitCode=0
Nov 02 22:07:51.962216 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:51.961849    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv" event={"ID":"923a65af-abd7-4774-93f9-94597d6cfb9f","Type":"ContainerDied","Data":"dfda084fcac97b09af08d47cfc92ef888c1f6d299f739bb803ec40de32203e32"}
Nov 02 22:07:53.157790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.157761    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:53.328740 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.328694    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-xks-ca\") pod \"923a65af-abd7-4774-93f9-94597d6cfb9f\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") "
Nov 02 22:07:53.329018 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.328994    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/923a65af-abd7-4774-93f9-94597d6cfb9f-kube-api-access-gz5lz\" (UniqueName: \"kubernetes.io/projected/923a65af-abd7-4774-93f9-94597d6cfb9f-kube-api-access-gz5lz\") pod \"923a65af-abd7-4774-93f9-94597d6cfb9f\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") "
Nov 02 22:07:53.329156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.329140    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-openshift-ca\") pod \"923a65af-abd7-4774-93f9-94597d6cfb9f\" (UID: \"923a65af-abd7-4774-93f9-94597d6cfb9f\") "
Nov 02 22:07:53.329867 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.329832    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-openshift-ca" pod "923a65af-abd7-4774-93f9-94597d6cfb9f" (UID: "923a65af-abd7-4774-93f9-94597d6cfb9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:07:53.331562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.331515    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-xks-ca" pod "923a65af-abd7-4774-93f9-94597d6cfb9f" (UID: "923a65af-abd7-4774-93f9-94597d6cfb9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:07:53.331831 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.331801    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923a65af-abd7-4774-93f9-94597d6cfb9f-kube-api-access-gz5lz" pod "923a65af-abd7-4774-93f9-94597d6cfb9f" (UID: "923a65af-abd7-4774-93f9-94597d6cfb9f"). InnerVolumeSpecName "kube-api-access-gz5lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:07:53.431335 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.431294    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:07:53.431516 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.431340    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923a65af-abd7-4774-93f9-94597d6cfb9f-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:07:53.431516 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.431356    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz5lz\" (UniqueName: \"kubernetes.io/projected/923a65af-abd7-4774-93f9-94597d6cfb9f-kube-api-access-gz5lz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:07:53.976371 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.976342    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"
Nov 02 22:07:53.976565 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.976376    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv" event={"ID":"923a65af-abd7-4774-93f9-94597d6cfb9f","Type":"ContainerDied","Data":"90a4fc79148b718f168fcae85e359d13b0723e1a0210567c348d86418bbf3557"}
Nov 02 22:07:53.976565 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:53.976425    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a4fc79148b718f168fcae85e359d13b0723e1a0210567c348d86418bbf3557"
Nov 02 22:07:57.047756 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:57.047725    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"]
Nov 02 22:07:57.097642 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:57.097612    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b8aecf-j8cjw"]
Nov 02 22:07:58.917669 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:07:58.917631    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb8d3a0-7eac-49d7-829f-50e5133987f1" path="/var/lib/kubelet/pods/bcb8d3a0-7eac-49d7-829f-50e5133987f1/volumes"
Nov 02 22:08:02.035160 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:02.035102    4528 generic.go:356] "Generic (PLEG): container finished" podID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerID="a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a" exitCode=0
Nov 02 22:08:02.035703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:02.035182    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" event={"ID":"505f97a2-29fe-4d06-a733-a182a3b87f03","Type":"ContainerDied","Data":"a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a"}
Nov 02 22:08:02.035703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:02.035617    4528 scope.go:122] "RemoveContainer" containerID="a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a"
Nov 02 22:08:02.037203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:02.037175    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:08:02.037280 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:02.037244    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:08:02.037280 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:02.037277    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:08:03.053350 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:03.053309    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" event={"ID":"505f97a2-29fe-4d06-a733-a182a3b87f03","Type":"ContainerStarted","Data":"d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061"}
Nov 02 22:08:03.079902 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:03.079874    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:08:03.080073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:03.080053    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 22:08:03.080138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:03.080100    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 22:08:04.080397 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:04.080366    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 22:08:04.080780 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:04.080413    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 22:08:05.080419 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:05.080386    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 22:08:05.080815 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:05.080446    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 22:08:06.079821 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:06.079785    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 22:08:06.080004 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:06.079844    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 22:08:07.079868 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:07.079834    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 22:08:07.080288 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:07.079881    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 22:08:08.080049 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:08.080008    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 02 22:08:08.080480 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:08.080071    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 02 22:08:09.085193 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:09.085160    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:08:09.085786 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:09.085707    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:08:09.086490 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:09.086466    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:08:14.114198 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:08:14.114152    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:08:14.897900 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:14.897854    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:08:23.207555 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:23.207515    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"]
Nov 02 22:08:23.313490 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:23.313452    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"]
Nov 02 22:08:28.208462 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.208418    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv" podUID="c0cc8b19-c952-46f2-a316-310cc3254e87" containerName="payload-processing" containerID="cri-o://05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef" gracePeriod=25
Nov 02 22:08:28.314362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.314310    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp" podUID="7c6d22ef-d7bb-42dd-acba-8a5479241c98" containerName="payload-pre-processing" containerID="cri-o://56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68" gracePeriod=25
Nov 02 22:08:28.479817 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.479748    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:08:28.502801 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.502773    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cc8b19-c952-46f2-a316-310cc3254e87-kube-api-access-rzx8p\" (UniqueName: \"kubernetes.io/projected/c0cc8b19-c952-46f2-a316-310cc3254e87-kube-api-access-rzx8p\") pod \"c0cc8b19-c952-46f2-a316-310cc3254e87\" (UID: \"c0cc8b19-c952-46f2-a316-310cc3254e87\") "
Nov 02 22:08:28.502979 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.502823    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0cc8b19-c952-46f2-a316-310cc3254e87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cc8b19-c952-46f2-a316-310cc3254e87-plugins-config-volume\") pod \"c0cc8b19-c952-46f2-a316-310cc3254e87\" (UID: \"c0cc8b19-c952-46f2-a316-310cc3254e87\") "
Nov 02 22:08:28.503272 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.503244    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cc8b19-c952-46f2-a316-310cc3254e87-plugins-config-volume" pod "c0cc8b19-c952-46f2-a316-310cc3254e87" (UID: "c0cc8b19-c952-46f2-a316-310cc3254e87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:08:28.505609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.505577    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cc8b19-c952-46f2-a316-310cc3254e87-kube-api-access-rzx8p" pod "c0cc8b19-c952-46f2-a316-310cc3254e87" (UID: "c0cc8b19-c952-46f2-a316-310cc3254e87"). InnerVolumeSpecName "kube-api-access-rzx8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:08:28.570447 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.570420    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:08:28.603640 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.603608    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6d22ef-d7bb-42dd-acba-8a5479241c98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6d22ef-d7bb-42dd-acba-8a5479241c98-plugins-config-volume\") pod \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\" (UID: \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\") "
Nov 02 22:08:28.603849 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.603770    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6d22ef-d7bb-42dd-acba-8a5479241c98-kube-api-access-8pg7c\" (UniqueName: \"kubernetes.io/projected/7c6d22ef-d7bb-42dd-acba-8a5479241c98-kube-api-access-8pg7c\") pod \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\" (UID: \"7c6d22ef-d7bb-42dd-acba-8a5479241c98\") "
Nov 02 22:08:28.604080 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.604040    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6d22ef-d7bb-42dd-acba-8a5479241c98-plugins-config-volume" pod "7c6d22ef-d7bb-42dd-acba-8a5479241c98" (UID: "7c6d22ef-d7bb-42dd-acba-8a5479241c98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:08:28.604400 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.604375    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzx8p\" (UniqueName: \"kubernetes.io/projected/c0cc8b19-c952-46f2-a316-310cc3254e87-kube-api-access-rzx8p\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:08:28.613201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.604404    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cc8b19-c952-46f2-a316-310cc3254e87-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:08:28.613201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.604418    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6d22ef-d7bb-42dd-acba-8a5479241c98-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:08:28.613201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.606053    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6d22ef-d7bb-42dd-acba-8a5479241c98-kube-api-access-8pg7c" pod "7c6d22ef-d7bb-42dd-acba-8a5479241c98" (UID: "7c6d22ef-d7bb-42dd-acba-8a5479241c98"). InnerVolumeSpecName "kube-api-access-8pg7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:08:28.725489 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:28.713359    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pg7c\" (UniqueName: \"kubernetes.io/projected/7c6d22ef-d7bb-42dd-acba-8a5479241c98-kube-api-access-8pg7c\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:08:29.228895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.228866    4528 generic.go:356] "Generic (PLEG): container finished" podID="c0cc8b19-c952-46f2-a316-310cc3254e87" containerID="05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef" exitCode=0
Nov 02 22:08:29.229354 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.228916    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"
Nov 02 22:08:29.229354 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.228950    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv" event={"ID":"c0cc8b19-c952-46f2-a316-310cc3254e87","Type":"ContainerDied","Data":"05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef"}
Nov 02 22:08:29.229354 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.229028    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv" event={"ID":"c0cc8b19-c952-46f2-a316-310cc3254e87","Type":"ContainerDied","Data":"bf3a9e206907811b0cb157c3afc6648464543430cfd9c596f3b3acdf1dc49580"}
Nov 02 22:08:29.229354 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.229058    4528 scope.go:122] "RemoveContainer" containerID="05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef"
Nov 02 22:08:29.230558 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.230530    4528 generic.go:356] "Generic (PLEG): container finished" podID="7c6d22ef-d7bb-42dd-acba-8a5479241c98" containerID="56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68" exitCode=0
Nov 02 22:08:29.230630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.230585    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"
Nov 02 22:08:29.230630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.230607    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp" event={"ID":"7c6d22ef-d7bb-42dd-acba-8a5479241c98","Type":"ContainerDied","Data":"56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68"}
Nov 02 22:08:29.230716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.230646    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp" event={"ID":"7c6d22ef-d7bb-42dd-acba-8a5479241c98","Type":"ContainerDied","Data":"aba1ac4286ad60eacb31d05671008e434c243b7fdc23acb628d1a5b371fee918"}
Nov 02 22:08:29.239646 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.239627    4528 scope.go:122] "RemoveContainer" containerID="05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef"
Nov 02 22:08:29.239928 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:08:29.239910    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef\": container with ID starting with 05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef not found: ID does not exist" containerID="05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef"
Nov 02 22:08:29.239981 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.239936    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef"} err="failed to get container status \"05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef\": rpc error: code = NotFound desc = could not find container \"05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef\": container with ID starting with 05719e794c0d5f7822b54c5948a1db55319e94966d9bfa1e36d8d6d6883d4aef not found: ID does not exist"
Nov 02 22:08:29.239981 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.239953    4528 scope.go:122] "RemoveContainer" containerID="56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68"
Nov 02 22:08:29.249739 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.249714    4528 scope.go:122] "RemoveContainer" containerID="56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68"
Nov 02 22:08:29.250024 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:08:29.250000    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68\": container with ID starting with 56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68 not found: ID does not exist" containerID="56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68"
Nov 02 22:08:29.250159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.250030    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68"} err="failed to get container status \"56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68\": rpc error: code = NotFound desc = could not find container \"56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68\": container with ID starting with 56feda80f6f6348eb3fec8167c18e5df895b5f8f129bca04dc769cbf27b1ec68 not found: ID does not exist"
Nov 02 22:08:29.394300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.394266    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"]
Nov 02 22:08:29.444171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.444140    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9981d2-6d4b6f758f-fdrjv"]
Nov 02 22:08:29.554353 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.554324    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"]
Nov 02 22:08:29.557516 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:29.557493    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9981d2-5487d48647-25tpp"]
Nov 02 22:08:30.914883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:30.914847    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6d22ef-d7bb-42dd-acba-8a5479241c98" path="/var/lib/kubelet/pods/7c6d22ef-d7bb-42dd-acba-8a5479241c98/volumes"
Nov 02 22:08:30.915348 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:30.915332    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0cc8b19-c952-46f2-a316-310cc3254e87" path="/var/lib/kubelet/pods/c0cc8b19-c952-46f2-a316-310cc3254e87/volumes"
Nov 02 22:08:35.308939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.308903    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"]
Nov 02 22:08:35.309558 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.309544    4528 cpu_manager.go:414] "containerMap: removing container" podUID="923a65af-abd7-4774-93f9-94597d6cfb9f" containerName="revoke-keys"
Nov 02 22:08:35.309610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.309559    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="923a65af-abd7-4774-93f9-94597d6cfb9f" containerName="revoke-keys"
Nov 02 22:08:35.309610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.309574    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7c6d22ef-d7bb-42dd-acba-8a5479241c98" containerName="payload-pre-processing"
Nov 02 22:08:35.309610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.309579    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6d22ef-d7bb-42dd-acba-8a5479241c98" containerName="payload-pre-processing"
Nov 02 22:08:35.309610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.309594    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c0cc8b19-c952-46f2-a316-310cc3254e87" containerName="payload-processing"
Nov 02 22:08:35.309610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.309599    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cc8b19-c952-46f2-a316-310cc3254e87" containerName="payload-processing"
Nov 02 22:08:35.309776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.309718    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="923a65af-abd7-4774-93f9-94597d6cfb9f" containerName="revoke-keys"
Nov 02 22:08:35.309776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.309728    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6d22ef-d7bb-42dd-acba-8a5479241c98" containerName="payload-pre-processing"
Nov 02 22:08:35.309776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.309734    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cc8b19-c952-46f2-a316-310cc3254e87" containerName="payload-processing"
Nov 02 22:08:35.314786 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.314766    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.372202 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.372166    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-363e15-openshift-default-dockercfg-fwr2t\""
Nov 02 22:08:35.416143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.416080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd786b7-e883-43a6-b539-c7c152512c0a-istiod-ca-cert\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.416329 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.416210    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-token\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.416329 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.416288    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-envoy\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.416460 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.416376    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-certs\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.416460 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.416428    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-socket\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.416574 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.416543    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-podinfo\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.416681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.416651    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-credential-socket\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.416832 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.416720    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-data\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.416832 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.416768    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdtz5\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-kube-api-access-vdtz5\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.427817 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.427792    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"]
Nov 02 22:08:35.518089 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518052    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-podinfo\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.518323 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518160    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-credential-socket\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.518323 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518196    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-data\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.518323 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518257    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdtz5\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-kube-api-access-vdtz5\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.518500 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518331    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd786b7-e883-43a6-b539-c7c152512c0a-istiod-ca-cert\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.518670 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518646    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-credential-socket\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.518766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518696    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-data\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.518766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518693    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-token\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.518882 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518813    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-envoy\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.518882 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518863    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-certs\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.519487 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.518901    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-socket\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.519487 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.519221    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-certs\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.519487 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.519285    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-socket\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.519487 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.519354    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd786b7-e883-43a6-b539-c7c152512c0a-istiod-ca-cert\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.520904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.520881    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-envoy\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.521422 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.521401    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-podinfo\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.575254 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.575198    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-token\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.575432 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.575413    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdtz5\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-kube-api-access-vdtz5\") pod \"e2e-autores-363e15-openshift-default-754479d777-k44vr\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") " pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:35.629142 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:35.629096    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:36.049860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:36.049798    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"]
Nov 02 22:08:36.082563 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:08:36.082531    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd786b7_e883_43a6_b539_c7c152512c0a.slice/crio-f474aa6cec389855250090319d74c35532f68ec9e45cf743b1f37f55e9f7f1cf WatchSource:0}: Error finding container f474aa6cec389855250090319d74c35532f68ec9e45cf743b1f37f55e9f7f1cf: Status 404 returned error can't find the container with id f474aa6cec389855250090319d74c35532f68ec9e45cf743b1f37f55e9f7f1cf
Nov 02 22:08:36.085254 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:36.085217    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:08:36.085389 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:36.085299    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:08:36.085389 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:36.085346    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:08:36.281657 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:36.281617    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" event={"ID":"bfd786b7-e883-43a6-b539-c7c152512c0a","Type":"ContainerStarted","Data":"cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96"}
Nov 02 22:08:36.281839 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:36.281683    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" event={"ID":"bfd786b7-e883-43a6-b539-c7c152512c0a","Type":"ContainerStarted","Data":"f474aa6cec389855250090319d74c35532f68ec9e45cf743b1f37f55e9f7f1cf"}
Nov 02 22:08:36.401630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:36.401525    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" podStartSLOduration=1.401508126 podStartE2EDuration="1.401508126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:36.398152543 +0000 UTC m=+3603.661688863" watchObservedRunningTime="2026-11-02 22:08:36.401508126 +0000 UTC m=+3603.665044445"
Nov 02 22:08:36.630071 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:36.630038    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:37.635868 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:37.635839    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:38.295519 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:38.295485    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:38.296292 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:38.296275    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:08:44.113947 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:08:44.113909    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:08:47.096339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.096302    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"]
Nov 02 22:08:47.101808 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.101788    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:47.201207 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.201169    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92879\" (UniqueName: \"kubernetes.io/projected/511810e8-c754-4efd-8224-991d37798c99-kube-api-access-92879\") pod \"payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l\" (UID: \"511810e8-c754-4efd-8224-991d37798c99\") " pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:47.201207 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.201221    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511810e8-c754-4efd-8224-991d37798c99-plugins-config-volume\") pod \"payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l\" (UID: \"511810e8-c754-4efd-8224-991d37798c99\") " pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:47.204654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.204626    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-363e15-dockercfg-6ndrh\""
Nov 02 22:08:47.204810 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.204677    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-363e15\""
Nov 02 22:08:47.208347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.208319    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"]
Nov 02 22:08:47.302417 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.302377    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92879\" (UniqueName: \"kubernetes.io/projected/511810e8-c754-4efd-8224-991d37798c99-kube-api-access-92879\") pod \"payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l\" (UID: \"511810e8-c754-4efd-8224-991d37798c99\") " pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:47.302591 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.302430    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511810e8-c754-4efd-8224-991d37798c99-plugins-config-volume\") pod \"payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l\" (UID: \"511810e8-c754-4efd-8224-991d37798c99\") " pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:47.303135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.303086    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511810e8-c754-4efd-8224-991d37798c99-plugins-config-volume\") pod \"payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l\" (UID: \"511810e8-c754-4efd-8224-991d37798c99\") " pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:47.360006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.359901    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92879\" (UniqueName: \"kubernetes.io/projected/511810e8-c754-4efd-8224-991d37798c99-kube-api-access-92879\") pod \"payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l\" (UID: \"511810e8-c754-4efd-8224-991d37798c99\") " pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:47.421630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.421594    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:47.791409 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:08:47.791378    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511810e8_c754_4efd_8224_991d37798c99.slice/crio-fe5bd71ae3ef00625906da05338fff6a5d73733cf1b6ae62f66bb8fe64991a52 WatchSource:0}: Error finding container fe5bd71ae3ef00625906da05338fff6a5d73733cf1b6ae62f66bb8fe64991a52: Status 404 returned error can't find the container with id fe5bd71ae3ef00625906da05338fff6a5d73733cf1b6ae62f66bb8fe64991a52
Nov 02 22:08:47.795987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:47.795960    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"]
Nov 02 22:08:48.364029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:48.363996    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l" event={"ID":"511810e8-c754-4efd-8224-991d37798c99","Type":"ContainerStarted","Data":"b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18"}
Nov 02 22:08:48.364549 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:48.364050    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l" event={"ID":"511810e8-c754-4efd-8224-991d37798c99","Type":"ContainerStarted","Data":"fe5bd71ae3ef00625906da05338fff6a5d73733cf1b6ae62f66bb8fe64991a52"}
Nov 02 22:08:48.364549 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:48.364199    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:48.489257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:48.489210    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l" podStartSLOduration=1.489195767 podStartE2EDuration="1.489195767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:48.476641282 +0000 UTC m=+3615.740177609" watchObservedRunningTime="2026-11-02 22:08:48.489195767 +0000 UTC m=+3615.752732085"
Nov 02 22:08:53.982012 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:53.981970    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"]
Nov 02 22:08:54.084871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:54.084832    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"]
Nov 02 22:08:56.049498 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:56.049466    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"]
Nov 02 22:08:56.099618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:56.099588    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c900efb-hfz7t"]
Nov 02 22:08:56.914169 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:56.914135    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4e238e-ab7f-438d-9684-06fce8976d49" path="/var/lib/kubelet/pods/bb4e238e-ab7f-438d-9684-06fce8976d49/volumes"
Nov 02 22:08:58.982540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:58.982483    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7" podUID="672f1f08-46db-4b28-934a-0e59117c6726" containerName="payload-processing" containerID="cri-o://020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a" gracePeriod=25
Nov 02 22:08:59.085418 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.085367    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8" podUID="12a7ced3-faf8-4292-ae0d-7109ae65a57a" containerName="payload-pre-processing" containerID="cri-o://de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438" gracePeriod=25
Nov 02 22:08:59.272406 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.272379    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:08:59.327310 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.327279    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672f1f08-46db-4b28-934a-0e59117c6726-kube-api-access-gmjrd\" (UniqueName: \"kubernetes.io/projected/672f1f08-46db-4b28-934a-0e59117c6726-kube-api-access-gmjrd\") pod \"672f1f08-46db-4b28-934a-0e59117c6726\" (UID: \"672f1f08-46db-4b28-934a-0e59117c6726\") "
Nov 02 22:08:59.327551 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.327455    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672f1f08-46db-4b28-934a-0e59117c6726-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f1f08-46db-4b28-934a-0e59117c6726-plugins-config-volume\") pod \"672f1f08-46db-4b28-934a-0e59117c6726\" (UID: \"672f1f08-46db-4b28-934a-0e59117c6726\") "
Nov 02 22:08:59.328220 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.328083    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672f1f08-46db-4b28-934a-0e59117c6726-plugins-config-volume" pod "672f1f08-46db-4b28-934a-0e59117c6726" (UID: "672f1f08-46db-4b28-934a-0e59117c6726"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:08:59.330716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.330693    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672f1f08-46db-4b28-934a-0e59117c6726-kube-api-access-gmjrd" pod "672f1f08-46db-4b28-934a-0e59117c6726" (UID: "672f1f08-46db-4b28-934a-0e59117c6726"). InnerVolumeSpecName "kube-api-access-gmjrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:08:59.375205 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.374305    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:08:59.388850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.388829    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:08:59.429004 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.428942    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a7ced3-faf8-4292-ae0d-7109ae65a57a-kube-api-access-dkf2n\" (UniqueName: \"kubernetes.io/projected/12a7ced3-faf8-4292-ae0d-7109ae65a57a-kube-api-access-dkf2n\") pod \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\" (UID: \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\") "
Nov 02 22:08:59.429230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.429208    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a7ced3-faf8-4292-ae0d-7109ae65a57a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7ced3-faf8-4292-ae0d-7109ae65a57a-plugins-config-volume\") pod \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\" (UID: \"12a7ced3-faf8-4292-ae0d-7109ae65a57a\") "
Nov 02 22:08:59.429587 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.429559    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a7ced3-faf8-4292-ae0d-7109ae65a57a-plugins-config-volume" pod "12a7ced3-faf8-4292-ae0d-7109ae65a57a" (UID: "12a7ced3-faf8-4292-ae0d-7109ae65a57a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:08:59.429921 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.429898    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjrd\" (UniqueName: \"kubernetes.io/projected/672f1f08-46db-4b28-934a-0e59117c6726-kube-api-access-gmjrd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:08:59.429921 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.429925    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7ced3-faf8-4292-ae0d-7109ae65a57a-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:08:59.430080 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.429939    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f1f08-46db-4b28-934a-0e59117c6726-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:08:59.431623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.431601    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a7ced3-faf8-4292-ae0d-7109ae65a57a-kube-api-access-dkf2n" pod "12a7ced3-faf8-4292-ae0d-7109ae65a57a" (UID: "12a7ced3-faf8-4292-ae0d-7109ae65a57a"). InnerVolumeSpecName "kube-api-access-dkf2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:08:59.499453 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.499374    4528 generic.go:356] "Generic (PLEG): container finished" podID="672f1f08-46db-4b28-934a-0e59117c6726" containerID="020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a" exitCode=0
Nov 02 22:08:59.499453 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.499433    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"
Nov 02 22:08:59.499697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.499458    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7" event={"ID":"672f1f08-46db-4b28-934a-0e59117c6726","Type":"ContainerDied","Data":"020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a"}
Nov 02 22:08:59.499697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.499511    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7" event={"ID":"672f1f08-46db-4b28-934a-0e59117c6726","Type":"ContainerDied","Data":"71a52cc39054ed38fde7c1f74c014636cc5f0f98c384e6c62d627a454f86f52b"}
Nov 02 22:08:59.499697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.499538    4528 scope.go:122] "RemoveContainer" containerID="020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a"
Nov 02 22:08:59.501305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.501272    4528 generic.go:356] "Generic (PLEG): container finished" podID="12a7ced3-faf8-4292-ae0d-7109ae65a57a" containerID="de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438" exitCode=0
Nov 02 22:08:59.501434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.501315    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8" event={"ID":"12a7ced3-faf8-4292-ae0d-7109ae65a57a","Type":"ContainerDied","Data":"de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438"}
Nov 02 22:08:59.501434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.501335    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"
Nov 02 22:08:59.501434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.501373    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8" event={"ID":"12a7ced3-faf8-4292-ae0d-7109ae65a57a","Type":"ContainerDied","Data":"5cfaa08f107a58324cbe5643f252e52002dfaa60b6b14e275bad615145c3ca5f"}
Nov 02 22:08:59.510225 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.510191    4528 scope.go:122] "RemoveContainer" containerID="020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a"
Nov 02 22:08:59.510581 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:08:59.510553    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a\": container with ID starting with 020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a not found: ID does not exist" containerID="020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a"
Nov 02 22:08:59.510707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.510587    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a"} err="failed to get container status \"020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a\": rpc error: code = NotFound desc = could not find container \"020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a\": container with ID starting with 020f7d7b2837202b159d3ab8b6c73844e19b6cb0159cc4970da0e077a07ec12a not found: ID does not exist"
Nov 02 22:08:59.510707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.510605    4528 scope.go:122] "RemoveContainer" containerID="de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438"
Nov 02 22:08:59.520729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.520712    4528 scope.go:122] "RemoveContainer" containerID="de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438"
Nov 02 22:08:59.521056 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:08:59.521036    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438\": container with ID starting with de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438 not found: ID does not exist" containerID="de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438"
Nov 02 22:08:59.521183 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.521061    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438"} err="failed to get container status \"de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438\": rpc error: code = NotFound desc = could not find container \"de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438\": container with ID starting with de840d30ae1ac6422840e21496bd76bc57a6453442a947e0fb5bf4e310fbc438 not found: ID does not exist"
Nov 02 22:08:59.530890 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.530862    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkf2n\" (UniqueName: \"kubernetes.io/projected/12a7ced3-faf8-4292-ae0d-7109ae65a57a-kube-api-access-dkf2n\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:08:59.741954 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.741923    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"]
Nov 02 22:08:59.746229 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.746190    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1538c4-74fdfdb7bc-6n4c8"]
Nov 02 22:08:59.855879 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.855849    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"]
Nov 02 22:08:59.859723 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:08:59.859702    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1538c4-58f9dd66b7-2qwg7"]
Nov 02 22:09:00.914256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:00.914227    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a7ced3-faf8-4292-ae0d-7109ae65a57a" path="/var/lib/kubelet/pods/12a7ced3-faf8-4292-ae0d-7109ae65a57a/volumes"
Nov 02 22:09:00.914649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:00.914624    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672f1f08-46db-4b28-934a-0e59117c6726" path="/var/lib/kubelet/pods/672f1f08-46db-4b28-934a-0e59117c6726/volumes"
Nov 02 22:09:07.050354 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:07.050316    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"]
Nov 02 22:09:07.098777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:07.098747    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df04840-xcldj"]
Nov 02 22:09:08.265926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:08.265897    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"]
Nov 02 22:09:08.266342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:08.266098    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy" containerID="cri-o://d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061" gracePeriod=30
Nov 02 22:09:08.915168 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:08.915135    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34604d88-db75-4517-9c28-8288012cbd92" path="/var/lib/kubelet/pods/34604d88-db75-4517-9c28-8288012cbd92/volumes"
Nov 02 22:09:09.088426 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:09.088381    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:09:09.088601 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:09.088449    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:09:13.544096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.544041    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:09:13.607337 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.607304    4528 generic.go:356] "Generic (PLEG): container finished" podID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerID="d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061" exitCode=0
Nov 02 22:09:13.607493 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.607358    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"
Nov 02 22:09:13.607493 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.607392    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" event={"ID":"505f97a2-29fe-4d06-a733-a182a3b87f03","Type":"ContainerDied","Data":"d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061"}
Nov 02 22:09:13.607493 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.607442    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx" event={"ID":"505f97a2-29fe-4d06-a733-a182a3b87f03","Type":"ContainerDied","Data":"887ef1053e88b72fac7ee36d9ae00416109847ecb4f361c3a8d3ed53e9d838dc"}
Nov 02 22:09:13.607493 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.607466    4528 scope.go:122] "RemoveContainer" containerID="d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061"
Nov 02 22:09:13.608341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.608319    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-envoy\") pod \"505f97a2-29fe-4d06-a733-a182a3b87f03\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") "
Nov 02 22:09:13.608464 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.608374    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-token\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-token\") pod \"505f97a2-29fe-4d06-a733-a182a3b87f03\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") "
Nov 02 22:09:13.608464 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.608414    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-socket\") pod \"505f97a2-29fe-4d06-a733-a182a3b87f03\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") "
Nov 02 22:09:13.608464 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.608455    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-podinfo\") pod \"505f97a2-29fe-4d06-a733-a182a3b87f03\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") "
Nov 02 22:09:13.608712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.608492    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505f97a2-29fe-4d06-a733-a182a3b87f03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505f97a2-29fe-4d06-a733-a182a3b87f03-istiod-ca-cert\") pod \"505f97a2-29fe-4d06-a733-a182a3b87f03\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") "
Nov 02 22:09:13.608712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.608538    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-kube-api-access-cdb57\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-kube-api-access-cdb57\") pod \"505f97a2-29fe-4d06-a733-a182a3b87f03\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") "
Nov 02 22:09:13.608712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.608568    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-credential-socket\") pod \"505f97a2-29fe-4d06-a733-a182a3b87f03\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") "
Nov 02 22:09:13.608712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.608597    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-certs\") pod \"505f97a2-29fe-4d06-a733-a182a3b87f03\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") "
Nov 02 22:09:13.608712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.608677    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-data\") pod \"505f97a2-29fe-4d06-a733-a182a3b87f03\" (UID: \"505f97a2-29fe-4d06-a733-a182a3b87f03\") "
Nov 02 22:09:13.609414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.609262    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-credential-socket" pod "505f97a2-29fe-4d06-a733-a182a3b87f03" (UID: "505f97a2-29fe-4d06-a733-a182a3b87f03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:09:13.609529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.609409    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:13.609529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.609438    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505f97a2-29fe-4d06-a733-a182a3b87f03-istiod-ca-cert" pod "505f97a2-29fe-4d06-a733-a182a3b87f03" (UID: "505f97a2-29fe-4d06-a733-a182a3b87f03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:09:13.609529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.609512    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-socket" pod "505f97a2-29fe-4d06-a733-a182a3b87f03" (UID: "505f97a2-29fe-4d06-a733-a182a3b87f03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:09:13.609736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.609660    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-data" pod "505f97a2-29fe-4d06-a733-a182a3b87f03" (UID: "505f97a2-29fe-4d06-a733-a182a3b87f03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:09:13.609736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.609718    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-certs" pod "505f97a2-29fe-4d06-a733-a182a3b87f03" (UID: "505f97a2-29fe-4d06-a733-a182a3b87f03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:09:13.610967 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.610939    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-envoy" pod "505f97a2-29fe-4d06-a733-a182a3b87f03" (UID: "505f97a2-29fe-4d06-a733-a182a3b87f03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:09:13.611294 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.611269    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-podinfo" pod "505f97a2-29fe-4d06-a733-a182a3b87f03" (UID: "505f97a2-29fe-4d06-a733-a182a3b87f03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:09:13.611384 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.611303    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-token" pod "505f97a2-29fe-4d06-a733-a182a3b87f03" (UID: "505f97a2-29fe-4d06-a733-a182a3b87f03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:09:13.611788 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.611765    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-kube-api-access-cdb57" pod "505f97a2-29fe-4d06-a733-a182a3b87f03" (UID: "505f97a2-29fe-4d06-a733-a182a3b87f03"). InnerVolumeSpecName "kube-api-access-cdb57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:09:13.638758 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.638740    4528 scope.go:122] "RemoveContainer" containerID="a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a"
Nov 02 22:09:13.648099 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.648081    4528 scope.go:122] "RemoveContainer" containerID="d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061"
Nov 02 22:09:13.648387 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:09:13.648369    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061\": container with ID starting with d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061 not found: ID does not exist" containerID="d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061"
Nov 02 22:09:13.648424 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.648394    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061"} err="failed to get container status \"d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061\": rpc error: code = NotFound desc = could not find container \"d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061\": container with ID starting with d639a55c8886bbc372e94c74643bcb6d7754d313303b11893509e3f3faea0061 not found: ID does not exist"
Nov 02 22:09:13.648424 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.648412    4528 scope.go:122] "RemoveContainer" containerID="a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a"
Nov 02 22:09:13.648674 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:09:13.648657    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a\": container with ID starting with a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a not found: ID does not exist" containerID="a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a"
Nov 02 22:09:13.648720 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.648677    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a"} err="failed to get container status \"a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a\": rpc error: code = NotFound desc = could not find container \"a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a\": container with ID starting with a971121c5fe9c7e69c1dc582ab3dbddd79fefb3bc0cebcd1c80ab6ae488e2d6a not found: ID does not exist"
Nov 02 22:09:13.710499 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.710471    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:13.710662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.710501    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:13.710662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.710516    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:13.710662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.710532    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:13.710662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.710549    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505f97a2-29fe-4d06-a733-a182a3b87f03-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:13.710662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.710575    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505f97a2-29fe-4d06-a733-a182a3b87f03-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:13.710662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.710591    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdb57\" (UniqueName: \"kubernetes.io/projected/505f97a2-29fe-4d06-a733-a182a3b87f03-kube-api-access-cdb57\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:13.710662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:13.710607    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505f97a2-29fe-4d06-a733-a182a3b87f03-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:14.029029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:14.028999    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"]
Nov 02 22:09:14.034508 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:14.034481    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1538c4-openshift-default-99ffc569-jjjkx"]
Nov 02 22:09:14.113470 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:09:14.113437    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:09:14.914548 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:14.914470    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" path="/var/lib/kubelet/pods/505f97a2-29fe-4d06-a733-a182a3b87f03/volumes"
Nov 02 22:09:15.934851 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.934815    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"]
Nov 02 22:09:15.935435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935421    4528 cpu_manager.go:414] "containerMap: removing container" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy"
Nov 02 22:09:15.935484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935436    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy"
Nov 02 22:09:15.935484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935457    4528 cpu_manager.go:414] "containerMap: removing container" podUID="12a7ced3-faf8-4292-ae0d-7109ae65a57a" containerName="payload-pre-processing"
Nov 02 22:09:15.935484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935462    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a7ced3-faf8-4292-ae0d-7109ae65a57a" containerName="payload-pre-processing"
Nov 02 22:09:15.935484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935478    4528 cpu_manager.go:414] "containerMap: removing container" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy"
Nov 02 22:09:15.935484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935484    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy"
Nov 02 22:09:15.935649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935503    4528 cpu_manager.go:414] "containerMap: removing container" podUID="672f1f08-46db-4b28-934a-0e59117c6726" containerName="payload-processing"
Nov 02 22:09:15.935649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935508    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672f1f08-46db-4b28-934a-0e59117c6726" containerName="payload-processing"
Nov 02 22:09:15.935649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935590    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy"
Nov 02 22:09:15.935649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935601    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a7ced3-faf8-4292-ae0d-7109ae65a57a" containerName="payload-pre-processing"
Nov 02 22:09:15.935649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935611    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="505f97a2-29fe-4d06-a733-a182a3b87f03" containerName="istio-proxy"
Nov 02 22:09:15.935649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.935618    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="672f1f08-46db-4b28-934a-0e59117c6726" containerName="payload-processing"
Nov 02 22:09:15.940409 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.940391    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:15.991252 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:15.991225    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"]
Nov 02 22:09:16.036773 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.036730    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-plugins-config-volume\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.036953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.036850    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-routing-overlay-volume\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.036953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.036937    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfrv\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-kube-api-access-vvfrv\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.037047 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.037011    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-provider-credentials\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.037820 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.037801    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 22:09:16.037963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.037944    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hlnch\""
Nov 02 22:09:16.037963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.037956    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 22:09:16.038069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.037984    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 22:09:16.043014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.042997    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 22:09:16.043099 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.042998    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-261b035ab1ee\""
Nov 02 22:09:16.138556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.138532    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-plugins-config-volume\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.138706 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.138586    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-routing-overlay-volume\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.138706 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.138616    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfrv\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-kube-api-access-vvfrv\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.138706 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.138659    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-provider-credentials\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.139484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.139461    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-plugins-config-volume\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.150582 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.150554    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-routing-overlay-volume\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.152840 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.152813    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-provider-credentials\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.206756 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.206698    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfrv\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-kube-api-access-vvfrv\") pod \"payload-processing-external-model-85864d6c5f-zqrgl\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") " pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.269411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.262994    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:16.688745 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:09:16.688710    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2e2272_961d_410c_8376_c7e863f3bcd7.slice/crio-d092ca1231993a4d716b5cbe6bdc9180f1f990f591eea5a3baad1cbe2e8e3893 WatchSource:0}: Error finding container d092ca1231993a4d716b5cbe6bdc9180f1f990f591eea5a3baad1cbe2e8e3893: Status 404 returned error can't find the container with id d092ca1231993a4d716b5cbe6bdc9180f1f990f591eea5a3baad1cbe2e8e3893
Nov 02 22:09:16.693685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:16.693656    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"]
Nov 02 22:09:17.637924 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:17.637886    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl" event={"ID":"ac2e2272-961d-410c-8376-c7e863f3bcd7","Type":"ContainerStarted","Data":"408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f"}
Nov 02 22:09:17.638422 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:17.637937    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl" event={"ID":"ac2e2272-961d-410c-8376-c7e863f3bcd7","Type":"ContainerStarted","Data":"d092ca1231993a4d716b5cbe6bdc9180f1f990f591eea5a3baad1cbe2e8e3893"}
Nov 02 22:09:17.638422 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:17.637991    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:17.754411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:17.754359    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl" podStartSLOduration=2.7543439320000003 podStartE2EDuration="2.754343932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:17.750835884 +0000 UTC m=+3645.014372213" watchObservedRunningTime="2026-11-02 22:09:17.754343932 +0000 UTC m=+3645.017880249"
Nov 02 22:09:26.810466 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:26.810430    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:09:28.648104 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:28.648074    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:09:35.730551 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:35.730522    4528 scope.go:122] "RemoveContainer" containerID="947b7f57d68e3a8723238cb83bd2d2567b649a4afa13f417fa1d15ee7227376f"
Nov 02 22:09:35.740464 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:35.740442    4528 scope.go:122] "RemoveContainer" containerID="214df7fa2338902e44084cdf21006e94da5d17033541db3b500cae87f7bf1dfd"
Nov 02 22:09:44.114932 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:09:44.114890    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:09:46.184761 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.184730    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"]
Nov 02 22:09:46.189692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.189659    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.245094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.245054    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.245094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.245123    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9s75\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-kube-api-access-w9s75\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.245357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.245213    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c92629-8829-478d-8f30-a732283d238c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.245357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.245246    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.245452 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.245368    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c92629-8829-478d-8f30-a732283d238c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.245452 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.245419    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.245577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.245478    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.245577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.245549    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.245694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.245603    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.312154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.311529    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"]
Nov 02 22:09:46.346508 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.346471    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c92629-8829-478d-8f30-a732283d238c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.346768 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.346540    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.346768 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.346566    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.346768 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.346591    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.346768 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.346650    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.346768 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.346723    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.347084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.346775    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9s75\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-kube-api-access-w9s75\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.347084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.346821    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c92629-8829-478d-8f30-a732283d238c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.347084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.346853    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.347315 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.347198    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.347513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.347491    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.347860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.347835    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.348274 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.348162    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c92629-8829-478d-8f30-a732283d238c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.348413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.348167    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.350059 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.350034    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.350923 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.350896    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c92629-8829-478d-8f30-a732283d238c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.355698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.355678    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4lw9\""
Nov 02 22:09:46.405453 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.405416    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.405614 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.405504    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9s75\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-kube-api-access-w9s75\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z62zb\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:46.514829 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:46.513421    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:47.003273 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:09:47.003237    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c92629_8829_478d_8f30_a732283d238c.slice/crio-71b1703e0af41dfed65f1d0f7fe9d9bb4fe3bad216d10e6808e608de7a37ba20 WatchSource:0}: Error finding container 71b1703e0af41dfed65f1d0f7fe9d9bb4fe3bad216d10e6808e608de7a37ba20: Status 404 returned error can't find the container with id 71b1703e0af41dfed65f1d0f7fe9d9bb4fe3bad216d10e6808e608de7a37ba20
Nov 02 22:09:47.006477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.006444    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"]
Nov 02 22:09:47.006606 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.006535    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:09:47.006682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.006619    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:09:47.006682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.006658    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:09:47.330424 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.330379    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"]
Nov 02 22:09:47.335960 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.335934    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.368238 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.368202    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ktp\" (UniqueName: \"kubernetes.io/projected/e99d66ec-4b6c-4c38-b3a3-34907e57185a-kube-api-access-m4ktp\") pod \"maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.368450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.368321    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.368450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.368372    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.384559 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.384522    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"]
Nov 02 22:09:47.393999 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.393888    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:09:47.393999 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.393929    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:09:47.393999 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.393972    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:09:47.394488 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.394469    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:09:47.469607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.469561    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ktp\" (UniqueName: \"kubernetes.io/projected/e99d66ec-4b6c-4c38-b3a3-34907e57185a-kube-api-access-m4ktp\") pod \"maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.469874 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.469658    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.469874 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.469687    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.473238 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.470669    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.473238 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.472634    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.527794 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.527756    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ktp\" (UniqueName: \"kubernetes.io/projected/e99d66ec-4b6c-4c38-b3a3-34907e57185a-kube-api-access-m4ktp\") pod \"maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.651276 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.651166    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:47.854267 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.854225    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" event={"ID":"99c92629-8829-478d-8f30-a732283d238c","Type":"ContainerStarted","Data":"6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371"}
Nov 02 22:09:47.854461 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.854290    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" event={"ID":"99c92629-8829-478d-8f30-a732283d238c","Type":"ContainerStarted","Data":"71b1703e0af41dfed65f1d0f7fe9d9bb4fe3bad216d10e6808e608de7a37ba20"}
Nov 02 22:09:47.993167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:47.993031    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" podStartSLOduration=1.9930139329999998 podStartE2EDuration="1.993013933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:47.991074437 +0000 UTC m=+3675.254610755" watchObservedRunningTime="2026-11-02 22:09:47.993013933 +0000 UTC m=+3675.256550251"
Nov 02 22:09:48.097603 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:09:48.097569    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99d66ec_4b6c_4c38_b3a3_34907e57185a.slice/crio-4e1bb2845c2a7ed29bd537830fb61a4bfea18a322d19efcad5ab3b3350955468 WatchSource:0}: Error finding container 4e1bb2845c2a7ed29bd537830fb61a4bfea18a322d19efcad5ab3b3350955468: Status 404 returned error can't find the container with id 4e1bb2845c2a7ed29bd537830fb61a4bfea18a322d19efcad5ab3b3350955468
Nov 02 22:09:48.098995 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:48.098971    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"]
Nov 02 22:09:48.526137 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:48.522424    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:48.528708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:48.528132    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z62zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:48.528708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:48.528205    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Nov 02 22:09:48.863224 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:48.863186    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464" event={"ID":"e99d66ec-4b6c-4c38-b3a3-34907e57185a","Type":"ContainerStarted","Data":"faa7889002a9396e406f60b812020dfce3722569f2490291ed1485898801225f"}
Nov 02 22:09:48.863428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:48.863254    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464" event={"ID":"e99d66ec-4b6c-4c38-b3a3-34907e57185a","Type":"ContainerStarted","Data":"4e1bb2845c2a7ed29bd537830fb61a4bfea18a322d19efcad5ab3b3350955468"}
Nov 02 22:09:48.989879 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:48.989814    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464" podStartSLOduration=1.989796304 podStartE2EDuration="1.989796304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:48.98679216 +0000 UTC m=+3676.250328472" watchObservedRunningTime="2026-11-02 22:09:48.989796304 +0000 UTC m=+3676.253332634"
Nov 02 22:09:49.297691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:49.297655    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"]
Nov 02 22:09:49.397263 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:49.397230    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"]
Nov 02 22:09:49.524131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:49.519513    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z62zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:49.524131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:49.519575    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Nov 02 22:09:49.880377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:49.880346    4528 generic.go:356] "Generic (PLEG): container finished" podID="e99d66ec-4b6c-4c38-b3a3-34907e57185a" containerID="faa7889002a9396e406f60b812020dfce3722569f2490291ed1485898801225f" exitCode=0
Nov 02 22:09:49.880810 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:49.880453    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464" event={"ID":"e99d66ec-4b6c-4c38-b3a3-34907e57185a","Type":"ContainerDied","Data":"faa7889002a9396e406f60b812020dfce3722569f2490291ed1485898801225f"}
Nov 02 22:09:50.526138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:50.521713    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z62zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:50.526138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:50.521792    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Nov 02 22:09:51.168702 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.168628    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:51.265901 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.265859    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-xks-ca\") pod \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") "
Nov 02 22:09:51.266075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.266028    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-openshift-ca\") pod \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") "
Nov 02 22:09:51.266164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.266145    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99d66ec-4b6c-4c38-b3a3-34907e57185a-kube-api-access-m4ktp\" (UniqueName: \"kubernetes.io/projected/e99d66ec-4b6c-4c38-b3a3-34907e57185a-kube-api-access-m4ktp\") pod \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\" (UID: \"e99d66ec-4b6c-4c38-b3a3-34907e57185a\") "
Nov 02 22:09:51.266380 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.266347    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-openshift-ca" pod "e99d66ec-4b6c-4c38-b3a3-34907e57185a" (UID: "e99d66ec-4b6c-4c38-b3a3-34907e57185a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:09:51.266658 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.266638    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:51.268031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.268001    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-xks-ca" pod "e99d66ec-4b6c-4c38-b3a3-34907e57185a" (UID: "e99d66ec-4b6c-4c38-b3a3-34907e57185a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:09:51.268370 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.268348    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99d66ec-4b6c-4c38-b3a3-34907e57185a-kube-api-access-m4ktp" pod "e99d66ec-4b6c-4c38-b3a3-34907e57185a" (UID: "e99d66ec-4b6c-4c38-b3a3-34907e57185a"). InnerVolumeSpecName "kube-api-access-m4ktp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:09:51.367600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.367562    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4ktp\" (UniqueName: \"kubernetes.io/projected/e99d66ec-4b6c-4c38-b3a3-34907e57185a-kube-api-access-m4ktp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:51.367600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.367598    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99d66ec-4b6c-4c38-b3a3-34907e57185a-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:51.520440 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.520400    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z62zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:51.520741 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.520706    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Nov 02 22:09:51.911208 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.901143    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464" event={"ID":"e99d66ec-4b6c-4c38-b3a3-34907e57185a","Type":"ContainerDied","Data":"4e1bb2845c2a7ed29bd537830fb61a4bfea18a322d19efcad5ab3b3350955468"}
Nov 02 22:09:51.920136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.911500    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e1bb2845c2a7ed29bd537830fb61a4bfea18a322d19efcad5ab3b3350955468"
Nov 02 22:09:51.922136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:51.921220    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"
Nov 02 22:09:52.523134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:52.522564    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z62zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:52.523134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:52.522664    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Nov 02 22:09:53.518937 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:53.518897    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z62zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:53.519130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:53.518966    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Nov 02 22:09:54.298846 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.298792    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz" podUID="d95af97f-15a0-4fc3-9804-e5a0b54e807d" containerName="payload-processing" containerID="cri-o://3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b" gracePeriod=25
Nov 02 22:09:54.397698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.397634    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44" podUID="bb9366e9-80af-4bc3-a179-d18ea766eab3" containerName="payload-pre-processing" containerID="cri-o://d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766" gracePeriod=25
Nov 02 22:09:54.519224 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.519075    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z62zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Nov 02 22:09:54.519918 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.519463    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Nov 02 22:09:54.609708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.609416    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:09:54.710215 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.710181    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95af97f-15a0-4fc3-9804-e5a0b54e807d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95af97f-15a0-4fc3-9804-e5a0b54e807d-plugins-config-volume\") pod \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\" (UID: \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\") "
Nov 02 22:09:54.710393 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.710373    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95af97f-15a0-4fc3-9804-e5a0b54e807d-kube-api-access-fnqc7\" (UniqueName: \"kubernetes.io/projected/d95af97f-15a0-4fc3-9804-e5a0b54e807d-kube-api-access-fnqc7\") pod \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\" (UID: \"d95af97f-15a0-4fc3-9804-e5a0b54e807d\") "
Nov 02 22:09:54.710613 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.710585    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95af97f-15a0-4fc3-9804-e5a0b54e807d-plugins-config-volume" pod "d95af97f-15a0-4fc3-9804-e5a0b54e807d" (UID: "d95af97f-15a0-4fc3-9804-e5a0b54e807d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:09:54.710906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.710885    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95af97f-15a0-4fc3-9804-e5a0b54e807d-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:54.712570 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.712538    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95af97f-15a0-4fc3-9804-e5a0b54e807d-kube-api-access-fnqc7" pod "d95af97f-15a0-4fc3-9804-e5a0b54e807d" (UID: "d95af97f-15a0-4fc3-9804-e5a0b54e807d"). InnerVolumeSpecName "kube-api-access-fnqc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:09:54.738396 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.738362    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:09:54.811958 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.811922    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9366e9-80af-4bc3-a179-d18ea766eab3-kube-api-access-qllwt\" (UniqueName: \"kubernetes.io/projected/bb9366e9-80af-4bc3-a179-d18ea766eab3-kube-api-access-qllwt\") pod \"bb9366e9-80af-4bc3-a179-d18ea766eab3\" (UID: \"bb9366e9-80af-4bc3-a179-d18ea766eab3\") "
Nov 02 22:09:54.812190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.812170    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb9366e9-80af-4bc3-a179-d18ea766eab3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9366e9-80af-4bc3-a179-d18ea766eab3-plugins-config-volume\") pod \"bb9366e9-80af-4bc3-a179-d18ea766eab3\" (UID: \"bb9366e9-80af-4bc3-a179-d18ea766eab3\") "
Nov 02 22:09:54.812515 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.812493    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9366e9-80af-4bc3-a179-d18ea766eab3-plugins-config-volume" pod "bb9366e9-80af-4bc3-a179-d18ea766eab3" (UID: "bb9366e9-80af-4bc3-a179-d18ea766eab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:09:54.812697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.812677    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnqc7\" (UniqueName: \"kubernetes.io/projected/d95af97f-15a0-4fc3-9804-e5a0b54e807d-kube-api-access-fnqc7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:54.812749 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.812706    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9366e9-80af-4bc3-a179-d18ea766eab3-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:54.814013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.813994    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9366e9-80af-4bc3-a179-d18ea766eab3-kube-api-access-qllwt" pod "bb9366e9-80af-4bc3-a179-d18ea766eab3" (UID: "bb9366e9-80af-4bc3-a179-d18ea766eab3"). InnerVolumeSpecName "kube-api-access-qllwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:09:54.914175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.914080    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qllwt\" (UniqueName: \"kubernetes.io/projected/bb9366e9-80af-4bc3-a179-d18ea766eab3-kube-api-access-qllwt\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:54.950736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.950700    4528 generic.go:356] "Generic (PLEG): container finished" podID="d95af97f-15a0-4fc3-9804-e5a0b54e807d" containerID="3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b" exitCode=0
Nov 02 22:09:54.950940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.950768    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"
Nov 02 22:09:54.950940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.950786    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz" event={"ID":"d95af97f-15a0-4fc3-9804-e5a0b54e807d","Type":"ContainerDied","Data":"3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b"}
Nov 02 22:09:54.950940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.950841    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz" event={"ID":"d95af97f-15a0-4fc3-9804-e5a0b54e807d","Type":"ContainerDied","Data":"e10a6d25ddb8aa4c611c30491b424121df7ebcb74895704ed43605c2c0e57ff0"}
Nov 02 22:09:54.950940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.950870    4528 scope.go:122] "RemoveContainer" containerID="3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b"
Nov 02 22:09:54.952498 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.952471    4528 generic.go:356] "Generic (PLEG): container finished" podID="bb9366e9-80af-4bc3-a179-d18ea766eab3" containerID="d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766" exitCode=0
Nov 02 22:09:54.952682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.952663    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44" event={"ID":"bb9366e9-80af-4bc3-a179-d18ea766eab3","Type":"ContainerDied","Data":"d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766"}
Nov 02 22:09:54.952744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.952715    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44" event={"ID":"bb9366e9-80af-4bc3-a179-d18ea766eab3","Type":"ContainerDied","Data":"5e83491e1c01654c798ac0f619969ed9abce8868f07b5b38cec1b29072b0738d"}
Nov 02 22:09:54.952799 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.952786    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"
Nov 02 22:09:54.962181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.962149    4528 scope.go:122] "RemoveContainer" containerID="3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b"
Nov 02 22:09:54.962526 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:09:54.962504    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b\": container with ID starting with 3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b not found: ID does not exist" containerID="3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b"
Nov 02 22:09:54.962605 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.962538    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b"} err="failed to get container status \"3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b\": rpc error: code = NotFound desc = could not find container \"3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b\": container with ID starting with 3c202428f3a68a0c76c4f0dbb08a9110686366fe98fdfc7dffc16c4387ebe95b not found: ID does not exist"
Nov 02 22:09:54.962605 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.962556    4528 scope.go:122] "RemoveContainer" containerID="d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766"
Nov 02 22:09:54.975609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.975471    4528 scope.go:122] "RemoveContainer" containerID="d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766"
Nov 02 22:09:54.975896 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:09:54.975866    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766\": container with ID starting with d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766 not found: ID does not exist" containerID="d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766"
Nov 02 22:09:54.975999 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:54.975904    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766"} err="failed to get container status \"d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766\": rpc error: code = NotFound desc = could not find container \"d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766\": container with ID starting with d1f258430e75b046cf767637991e75e6a7fbbced5b96c8cf32b69bc25ee6b766 not found: ID does not exist"
Nov 02 22:09:55.119966 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.119917    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"]
Nov 02 22:09:55.132135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.123929    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8702a9-68bdf96c7-bt4lz"]
Nov 02 22:09:55.230970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.230876    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"]
Nov 02 22:09:55.234377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.234341    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8702a9-5b98568795-nzg44"]
Nov 02 22:09:55.497455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.497344    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"]
Nov 02 22:09:55.499079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.499047    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d95af97f-15a0-4fc3-9804-e5a0b54e807d" containerName="payload-processing"
Nov 02 22:09:55.499274 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.499258    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95af97f-15a0-4fc3-9804-e5a0b54e807d" containerName="payload-processing"
Nov 02 22:09:55.499401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.499388    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bb9366e9-80af-4bc3-a179-d18ea766eab3" containerName="payload-pre-processing"
Nov 02 22:09:55.499506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.499492    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9366e9-80af-4bc3-a179-d18ea766eab3" containerName="payload-pre-processing"
Nov 02 22:09:55.499651 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.499638    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e99d66ec-4b6c-4c38-b3a3-34907e57185a" containerName="revoke-keys"
Nov 02 22:09:55.499742 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.499731    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99d66ec-4b6c-4c38-b3a3-34907e57185a" containerName="revoke-keys"
Nov 02 22:09:55.499997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.499979    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99d66ec-4b6c-4c38-b3a3-34907e57185a" containerName="revoke-keys"
Nov 02 22:09:55.500156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.500142    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95af97f-15a0-4fc3-9804-e5a0b54e807d" containerName="payload-processing"
Nov 02 22:09:55.500273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.500260    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9366e9-80af-4bc3-a179-d18ea766eab3" containerName="payload-pre-processing"
Nov 02 22:09:55.509474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.509441    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.555148 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.553210    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"]
Nov 02 22:09:55.564509 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.564475    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:09:55.564987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.564962    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:09:55.565807 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.565786    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:09:55.604707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.604671    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:09:55.635415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.635371    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjmmh\" (UniqueName: \"kubernetes.io/projected/fc02305c-98ad-4d66-b8d0-8955d6d251c7-kube-api-access-fjmmh\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.636873 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.636845    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.648149 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.646102    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.751060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.750124    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjmmh\" (UniqueName: \"kubernetes.io/projected/fc02305c-98ad-4d66-b8d0-8955d6d251c7-kube-api-access-fjmmh\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.751060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.750223    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.751060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.750400    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.752508 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.752482    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.767894 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.767829    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.823129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.818354    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjmmh\" (UniqueName: \"kubernetes.io/projected/fc02305c-98ad-4d66-b8d0-8955d6d251c7-kube-api-access-fjmmh\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:55.854130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:55.853761    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:56.053131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.052326    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"]
Nov 02 22:09:56.061157 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.058434    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3785705d-l4fhj"]
Nov 02 22:09:56.207162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.204421    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:56.207162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.206204    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:56.214731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.211940    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:09:56.316217 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:09:56.316176    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc02305c_98ad_4d66_b8d0_8955d6d251c7.slice/crio-4083b8a616a8f320a64b6da7177985e71263d6c6a66d01f590c97c580988449a WatchSource:0}: Error finding container 4083b8a616a8f320a64b6da7177985e71263d6c6a66d01f590c97c580988449a: Status 404 returned error can't find the container with id 4083b8a616a8f320a64b6da7177985e71263d6c6a66d01f590c97c580988449a
Nov 02 22:09:56.319612 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.319583    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"]
Nov 02 22:09:56.914732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.914695    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907c8310-2e6e-4ce4-8a52-0c08a77ff0c8" path="/var/lib/kubelet/pods/907c8310-2e6e-4ce4-8a52-0c08a77ff0c8/volumes"
Nov 02 22:09:56.915131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.915100    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9366e9-80af-4bc3-a179-d18ea766eab3" path="/var/lib/kubelet/pods/bb9366e9-80af-4bc3-a179-d18ea766eab3/volumes"
Nov 02 22:09:56.915497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.915484    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95af97f-15a0-4fc3-9804-e5a0b54e807d" path="/var/lib/kubelet/pods/d95af97f-15a0-4fc3-9804-e5a0b54e807d/volumes"
Nov 02 22:09:56.985079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.984994    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8" event={"ID":"fc02305c-98ad-4d66-b8d0-8955d6d251c7","Type":"ContainerStarted","Data":"bb9bfca4a0fa8b4df909b264ae88818e709edebac0a4818f7ab4dce998e7ddbd"}
Nov 02 22:09:56.985079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:56.985049    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8" event={"ID":"fc02305c-98ad-4d66-b8d0-8955d6d251c7","Type":"ContainerStarted","Data":"4083b8a616a8f320a64b6da7177985e71263d6c6a66d01f590c97c580988449a"}
Nov 02 22:09:57.098953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:57.098899    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8" podStartSLOduration=2.098884224 podStartE2EDuration="2.098884224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:57.097335931 +0000 UTC m=+3684.360872250" watchObservedRunningTime="2026-11-02 22:09:57.098884224 +0000 UTC m=+3684.362420545"
Nov 02 22:09:57.993568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:57.993526    4528 generic.go:356] "Generic (PLEG): container finished" podID="fc02305c-98ad-4d66-b8d0-8955d6d251c7" containerID="bb9bfca4a0fa8b4df909b264ae88818e709edebac0a4818f7ab4dce998e7ddbd" exitCode=0
Nov 02 22:09:57.994029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:57.993610    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8" event={"ID":"fc02305c-98ad-4d66-b8d0-8955d6d251c7","Type":"ContainerDied","Data":"bb9bfca4a0fa8b4df909b264ae88818e709edebac0a4818f7ab4dce998e7ddbd"}
Nov 02 22:09:58.517940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.517900    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"]
Nov 02 22:09:58.522221 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.522197    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.574925 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.574892    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"]
Nov 02 22:09:58.609218 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.609184    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad1d0fb-8ctpk\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.609408 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.609257    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trzjj\" (UniqueName: \"kubernetes.io/projected/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-kube-api-access-trzjj\") pod \"maas-api-revoke-sub-bad1d0fb-8ctpk\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.609408 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.609346    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad1d0fb-8ctpk\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.710676 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.710634    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad1d0fb-8ctpk\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.710880 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.710721    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trzjj\" (UniqueName: \"kubernetes.io/projected/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-kube-api-access-trzjj\") pod \"maas-api-revoke-sub-bad1d0fb-8ctpk\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.710880 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.710750    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad1d0fb-8ctpk\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.711493 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.711472    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad1d0fb-8ctpk\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.712953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.712934    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad1d0fb-8ctpk\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.767716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.767681    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trzjj\" (UniqueName: \"kubernetes.io/projected/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-kube-api-access-trzjj\") pod \"maas-api-revoke-sub-bad1d0fb-8ctpk\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:58.836423 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:58.836339    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:09:59.136324 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.136296    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:09:59.208807 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:09:59.208759    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1b16e2_ccb4_4290_a0c1_e60810f6ac4a.slice/crio-e85410825b717967499deaf471c439ca48e4c7d04f530ab82cf5d826255c56f4 WatchSource:0}: Error finding container e85410825b717967499deaf471c439ca48e4c7d04f530ab82cf5d826255c56f4: Status 404 returned error can't find the container with id e85410825b717967499deaf471c439ca48e4c7d04f530ab82cf5d826255c56f4
Nov 02 22:09:59.216942 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.216913    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-openshift-ca\") pod \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") "
Nov 02 22:09:59.217072 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.217004    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-xks-ca\") pod \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") "
Nov 02 22:09:59.217072 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.217030    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc02305c-98ad-4d66-b8d0-8955d6d251c7-kube-api-access-fjmmh\" (UniqueName: \"kubernetes.io/projected/fc02305c-98ad-4d66-b8d0-8955d6d251c7-kube-api-access-fjmmh\") pod \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\" (UID: \"fc02305c-98ad-4d66-b8d0-8955d6d251c7\") "
Nov 02 22:09:59.217277 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.217254    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-openshift-ca" pod "fc02305c-98ad-4d66-b8d0-8955d6d251c7" (UID: "fc02305c-98ad-4d66-b8d0-8955d6d251c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:09:59.217594 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.217568    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:59.218988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.218964    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-xks-ca" pod "fc02305c-98ad-4d66-b8d0-8955d6d251c7" (UID: "fc02305c-98ad-4d66-b8d0-8955d6d251c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:09:59.219138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.219089    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc02305c-98ad-4d66-b8d0-8955d6d251c7-kube-api-access-fjmmh" pod "fc02305c-98ad-4d66-b8d0-8955d6d251c7" (UID: "fc02305c-98ad-4d66-b8d0-8955d6d251c7"). InnerVolumeSpecName "kube-api-access-fjmmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:09:59.241920 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.241887    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"]
Nov 02 22:09:59.319229 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.319189    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc02305c-98ad-4d66-b8d0-8955d6d251c7-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:09:59.319229 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:09:59.319223    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjmmh\" (UniqueName: \"kubernetes.io/projected/fc02305c-98ad-4d66-b8d0-8955d6d251c7-kube-api-access-fjmmh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:00.007603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:00.007521    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"
Nov 02 22:10:00.007603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:00.007543    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8" event={"ID":"fc02305c-98ad-4d66-b8d0-8955d6d251c7","Type":"ContainerDied","Data":"4083b8a616a8f320a64b6da7177985e71263d6c6a66d01f590c97c580988449a"}
Nov 02 22:10:00.007836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:00.007603    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4083b8a616a8f320a64b6da7177985e71263d6c6a66d01f590c97c580988449a"
Nov 02 22:10:00.009460 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:00.009430    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk" event={"ID":"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a","Type":"ContainerStarted","Data":"cac116d4761883d7a008d85a04188e543773a8d9f2ef1beb7e2e04f0f5e92671"}
Nov 02 22:10:00.009609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:00.009480    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk" event={"ID":"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a","Type":"ContainerStarted","Data":"e85410825b717967499deaf471c439ca48e4c7d04f530ab82cf5d826255c56f4"}
Nov 02 22:10:00.125703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:00.125656    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk" podStartSLOduration=2.125641188 podStartE2EDuration="2.125641188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:00.123391999 +0000 UTC m=+3687.386928317" watchObservedRunningTime="2026-11-02 22:10:00.125641188 +0000 UTC m=+3687.389177533"
Nov 02 22:10:01.017838 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:01.017803    4528 generic.go:356] "Generic (PLEG): container finished" podID="1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a" containerID="cac116d4761883d7a008d85a04188e543773a8d9f2ef1beb7e2e04f0f5e92671" exitCode=0
Nov 02 22:10:01.018280 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:01.017893    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk" event={"ID":"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a","Type":"ContainerDied","Data":"cac116d4761883d7a008d85a04188e543773a8d9f2ef1beb7e2e04f0f5e92671"}
Nov 02 22:10:02.168350 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.168326    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:10:02.253795 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.253762    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-xks-ca\") pod \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") "
Nov 02 22:10:02.253971 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.253954    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-openshift-ca\") pod \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") "
Nov 02 22:10:02.254019 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.253997    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-kube-api-access-trzjj\" (UniqueName: \"kubernetes.io/projected/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-kube-api-access-trzjj\") pod \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\" (UID: \"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a\") "
Nov 02 22:10:02.254337 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.254311    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-openshift-ca" pod "1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a" (UID: "1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:10:02.254551 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.254532    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:02.256002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.255978    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-xks-ca" pod "1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a" (UID: "1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:10:02.256102 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.256083    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-kube-api-access-trzjj" pod "1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a" (UID: "1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a"). InnerVolumeSpecName "kube-api-access-trzjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:10:02.355492 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.355412    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:02.355492 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:02.355439    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trzjj\" (UniqueName: \"kubernetes.io/projected/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a-kube-api-access-trzjj\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:03.033334 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:03.033305    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"
Nov 02 22:10:03.033538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:03.033341    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk" event={"ID":"1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a","Type":"ContainerDied","Data":"e85410825b717967499deaf471c439ca48e4c7d04f530ab82cf5d826255c56f4"}
Nov 02 22:10:03.033538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:03.033410    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e85410825b717967499deaf471c439ca48e4c7d04f530ab82cf5d826255c56f4"
Nov 02 22:10:07.046544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:07.046509    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"]
Nov 02 22:10:07.096576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:07.096543    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45e7e934-q5749"]
Nov 02 22:10:08.915147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:08.915101    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea41520-417b-475d-8c6e-c8343694fa58" path="/var/lib/kubelet/pods/4ea41520-417b-475d-8c6e-c8343694fa58/volumes"
Nov 02 22:10:10.207401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:10.207370    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:10:10.610058 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:10.610025    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"]
Nov 02 22:10:14.115811 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:14.115768    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:10:15.610495 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.610454    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl" podUID="ac2e2272-961d-410c-8376-c7e863f3bcd7" containerName="payload-processing" containerID="cri-o://408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f" gracePeriod=25
Nov 02 22:10:15.869162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.869077    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:10:15.963945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.963911    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-routing-overlay-volume\") pod \"ac2e2272-961d-410c-8376-c7e863f3bcd7\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") "
Nov 02 22:10:15.964180 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.963966    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-provider-credentials\") pod \"ac2e2272-961d-410c-8376-c7e863f3bcd7\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") "
Nov 02 22:10:15.964180 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.964105    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-plugins-config-volume\") pod \"ac2e2272-961d-410c-8376-c7e863f3bcd7\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") "
Nov 02 22:10:15.964331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.964201    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-kube-api-access-vvfrv\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-kube-api-access-vvfrv\") pod \"ac2e2272-961d-410c-8376-c7e863f3bcd7\" (UID: \"ac2e2272-961d-410c-8376-c7e863f3bcd7\") "
Nov 02 22:10:15.964460 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.964427    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-routing-overlay-volume" pod "ac2e2272-961d-410c-8376-c7e863f3bcd7" (UID: "ac2e2272-961d-410c-8376-c7e863f3bcd7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:10:15.964596 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.964573    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-plugins-config-volume" pod "ac2e2272-961d-410c-8376-c7e863f3bcd7" (UID: "ac2e2272-961d-410c-8376-c7e863f3bcd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:10:15.964676 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.964656    4528 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-routing-overlay-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:15.966191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.966171    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-provider-credentials" pod "ac2e2272-961d-410c-8376-c7e863f3bcd7" (UID: "ac2e2272-961d-410c-8376-c7e863f3bcd7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:10:15.966492 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:15.966472    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-kube-api-access-vvfrv" pod "ac2e2272-961d-410c-8376-c7e863f3bcd7" (UID: "ac2e2272-961d-410c-8376-c7e863f3bcd7"). InnerVolumeSpecName "kube-api-access-vvfrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:10:16.065897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.065862    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2e2272-961d-410c-8376-c7e863f3bcd7-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:16.065897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.065890    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvfrv\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-kube-api-access-vvfrv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:16.065897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.065900    4528 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac2e2272-961d-410c-8376-c7e863f3bcd7-provider-credentials\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:16.158264 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.158173    4528 generic.go:356] "Generic (PLEG): container finished" podID="ac2e2272-961d-410c-8376-c7e863f3bcd7" containerID="408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f" exitCode=0
Nov 02 22:10:16.158264 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.158228    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"
Nov 02 22:10:16.158264 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.158246    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl" event={"ID":"ac2e2272-961d-410c-8376-c7e863f3bcd7","Type":"ContainerDied","Data":"408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f"}
Nov 02 22:10:16.158547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.158303    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl" event={"ID":"ac2e2272-961d-410c-8376-c7e863f3bcd7","Type":"ContainerDied","Data":"d092ca1231993a4d716b5cbe6bdc9180f1f990f591eea5a3baad1cbe2e8e3893"}
Nov 02 22:10:16.158547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.158327    4528 scope.go:122] "RemoveContainer" containerID="408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f"
Nov 02 22:10:16.168701 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.168682    4528 scope.go:122] "RemoveContainer" containerID="408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f"
Nov 02 22:10:16.168984 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:16.168966    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f\": container with ID starting with 408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f not found: ID does not exist" containerID="408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f"
Nov 02 22:10:16.169029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.168991    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f"} err="failed to get container status \"408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f\": rpc error: code = NotFound desc = could not find container \"408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f\": container with ID starting with 408e4d0aa6e50e1403c2cf8765a959b0c0c3b7d9aeb4085da37af89f6c971c5f not found: ID does not exist"
Nov 02 22:10:16.325790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.325749    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"]
Nov 02 22:10:16.332007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.331983    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85864d6c5f-zqrgl"]
Nov 02 22:10:16.915545 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.915516    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2e2272-961d-410c-8376-c7e863f3bcd7" path="/var/lib/kubelet/pods/ac2e2272-961d-410c-8376-c7e863f3bcd7/volumes"
Nov 02 22:10:16.917019 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.916994    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"]
Nov 02 22:10:16.917592 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.917579    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fc02305c-98ad-4d66-b8d0-8955d6d251c7" containerName="revoke-keys"
Nov 02 22:10:16.917642 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.917594    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc02305c-98ad-4d66-b8d0-8955d6d251c7" containerName="revoke-keys"
Nov 02 22:10:16.917642 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.917606    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a" containerName="revoke-keys"
Nov 02 22:10:16.917642 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.917612    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a" containerName="revoke-keys"
Nov 02 22:10:16.917642 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.917634    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ac2e2272-961d-410c-8376-c7e863f3bcd7" containerName="payload-processing"
Nov 02 22:10:16.917642 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.917639    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2e2272-961d-410c-8376-c7e863f3bcd7" containerName="payload-processing"
Nov 02 22:10:16.917792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.917725    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc02305c-98ad-4d66-b8d0-8955d6d251c7" containerName="revoke-keys"
Nov 02 22:10:16.917792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.917733    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a" containerName="revoke-keys"
Nov 02 22:10:16.917792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.917744    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2e2272-961d-410c-8376-c7e863f3bcd7" containerName="payload-processing"
Nov 02 22:10:16.922676 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.922658    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:16.973708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.973673    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"]
Nov 02 22:10:16.980703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.980675    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-routing-overlay-volume\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:16.980891 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.980755    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-plugins-config-volume\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:16.981017 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.980990    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:16.981084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:16.981037    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lm7h\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-kube-api-access-9lm7h\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.019333 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.019305    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3634f8f9043c\""
Nov 02 22:10:17.019333 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.019315    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 22:10:17.019556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.019360    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 22:10:17.019556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.019379    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 22:10:17.019556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.019406    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 22:10:17.025504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.025480    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4f522\""
Nov 02 22:10:17.082237 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.082198    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-plugins-config-volume\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.082414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.082314    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.082414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.082344    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lm7h\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-kube-api-access-9lm7h\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.082540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.082445    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-routing-overlay-volume\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.083018 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.082993    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-routing-overlay-volume\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.083137 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.083023    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-plugins-config-volume\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.084807 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.084782    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.139689 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.139655    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lm7h\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-kube-api-access-9lm7h\") pod \"payload-processing-external-model-6f79fb9565-k7p7n\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.250182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.240434    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:17.674195 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:10:17.674164    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c450dbd_b4cf_4f19_a98f_379bfab0ce70.slice/crio-59ed23b4269da80fd41c9812ac7db702eed2137a09e2d2f1256b6cab9d646ec2 WatchSource:0}: Error finding container 59ed23b4269da80fd41c9812ac7db702eed2137a09e2d2f1256b6cab9d646ec2: Status 404 returned error can't find the container with id 59ed23b4269da80fd41c9812ac7db702eed2137a09e2d2f1256b6cab9d646ec2
Nov 02 22:10:17.676961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:17.676929    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"]
Nov 02 22:10:18.175510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:18.175460    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n" event={"ID":"4c450dbd-b4cf-4f19-a98f-379bfab0ce70","Type":"ContainerStarted","Data":"3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b"}
Nov 02 22:10:18.175942 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:18.175522    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n" event={"ID":"4c450dbd-b4cf-4f19-a98f-379bfab0ce70","Type":"ContainerStarted","Data":"59ed23b4269da80fd41c9812ac7db702eed2137a09e2d2f1256b6cab9d646ec2"}
Nov 02 22:10:18.175942 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:18.175560    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:18.292641 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:18.292589    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n" podStartSLOduration=2.292575544 podStartE2EDuration="2.292575544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:18.289718076 +0000 UTC m=+3705.553254393" watchObservedRunningTime="2026-11-02 22:10:18.292575544 +0000 UTC m=+3705.556111883"
Nov 02 22:10:19.211238 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:19.211208    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3634f8f9043c: secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:19.211238 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:19.211235    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n: secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:19.211709 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:19.211293    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials podName:4c450dbd-b4cf-4f19-a98f-379bfab0ce70 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:19.711273027 +0000 UTC m=+3706.974809346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials") pod "payload-processing-external-model-6f79fb9565-k7p7n" (UID: "4c450dbd-b4cf-4f19-a98f-379bfab0ce70") : secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:19.718266 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:19.718232    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3634f8f9043c: secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:19.718266 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:19.718259    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n: secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:19.718503 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:19.718314    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials podName:4c450dbd-b4cf-4f19-a98f-379bfab0ce70 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:20.718299908 +0000 UTC m=+3707.981836227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials") pod "payload-processing-external-model-6f79fb9565-k7p7n" (UID: "4c450dbd-b4cf-4f19-a98f-379bfab0ce70") : secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:20.065464 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:20.065427    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"]
Nov 02 22:10:20.740307 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:20.740261    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3634f8f9043c: secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:20.740873 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:20.740852    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n: secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:20.741050 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:20.741037    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials podName:4c450dbd-b4cf-4f19-a98f-379bfab0ce70 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:22.741011878 +0000 UTC m=+3710.004548188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials") pod "payload-processing-external-model-6f79fb9565-k7p7n" (UID: "4c450dbd-b4cf-4f19-a98f-379bfab0ce70") : secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:22.770485 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:22.770450    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3634f8f9043c: secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:22.770485 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:22.770478    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n: secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:22.770925 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:22.770531    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials podName:4c450dbd-b4cf-4f19-a98f-379bfab0ce70 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:26.77051865 +0000 UTC m=+3714.034054946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials") pod "payload-processing-external-model-6f79fb9565-k7p7n" (UID: "4c450dbd-b4cf-4f19-a98f-379bfab0ce70") : secret "agc-cred-3634f8f9043c" not found
Nov 02 22:10:25.190202 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.190152    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n" podUID="4c450dbd-b4cf-4f19-a98f-379bfab0ce70" containerName="payload-processing" containerID="cri-o://3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b" gracePeriod=25
Nov 02 22:10:25.472390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.472364    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:25.628694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.628656    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-plugins-config-volume\") pod \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") "
Nov 02 22:10:25.628887 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.628710    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials\") pod \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") "
Nov 02 22:10:25.628887 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.628836    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-routing-overlay-volume\") pod \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") "
Nov 02 22:10:25.629020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.628937    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-kube-api-access-9lm7h\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-kube-api-access-9lm7h\") pod \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\" (UID: \"4c450dbd-b4cf-4f19-a98f-379bfab0ce70\") "
Nov 02 22:10:25.629204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.629157    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-plugins-config-volume" pod "4c450dbd-b4cf-4f19-a98f-379bfab0ce70" (UID: "4c450dbd-b4cf-4f19-a98f-379bfab0ce70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:10:25.629204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.629188    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-routing-overlay-volume" pod "4c450dbd-b4cf-4f19-a98f-379bfab0ce70" (UID: "4c450dbd-b4cf-4f19-a98f-379bfab0ce70"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:10:25.629544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.629524    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:25.629544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.629546    4528 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-routing-overlay-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:25.630939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.630913    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials" pod "4c450dbd-b4cf-4f19-a98f-379bfab0ce70" (UID: "4c450dbd-b4cf-4f19-a98f-379bfab0ce70"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:10:25.631066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.630963    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-kube-api-access-9lm7h" pod "4c450dbd-b4cf-4f19-a98f-379bfab0ce70" (UID: "4c450dbd-b4cf-4f19-a98f-379bfab0ce70"). InnerVolumeSpecName "kube-api-access-9lm7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:10:25.732295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.732070    4528 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-provider-credentials\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:25.732295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:25.732102    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lm7h\" (UniqueName: \"kubernetes.io/projected/4c450dbd-b4cf-4f19-a98f-379bfab0ce70-kube-api-access-9lm7h\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:26.230827 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.230794    4528 generic.go:356] "Generic (PLEG): container finished" podID="4c450dbd-b4cf-4f19-a98f-379bfab0ce70" containerID="3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b" exitCode=0
Nov 02 22:10:26.231299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.230845    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"
Nov 02 22:10:26.231299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.230877    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n" event={"ID":"4c450dbd-b4cf-4f19-a98f-379bfab0ce70","Type":"ContainerDied","Data":"3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b"}
Nov 02 22:10:26.231299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.230937    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n" event={"ID":"4c450dbd-b4cf-4f19-a98f-379bfab0ce70","Type":"ContainerDied","Data":"59ed23b4269da80fd41c9812ac7db702eed2137a09e2d2f1256b6cab9d646ec2"}
Nov 02 22:10:26.231299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.230965    4528 scope.go:122] "RemoveContainer" containerID="3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b"
Nov 02 22:10:26.242288 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.242267    4528 scope.go:122] "RemoveContainer" containerID="3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b"
Nov 02 22:10:26.242574 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:26.242555    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b\": container with ID starting with 3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b not found: ID does not exist" containerID="3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b"
Nov 02 22:10:26.242627 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.242584    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b"} err="failed to get container status \"3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b\": rpc error: code = NotFound desc = could not find container \"3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b\": container with ID starting with 3fbf91ea4879ab709d8b73aebda23af74e82ff9ebcf1657fd54bb2e976f0ad8b not found: ID does not exist"
Nov 02 22:10:26.398799 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.398766    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"]
Nov 02 22:10:26.448607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.448578    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-k7p7n"]
Nov 02 22:10:26.914510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:26.914471    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c450dbd-b4cf-4f19-a98f-379bfab0ce70" path="/var/lib/kubelet/pods/4c450dbd-b4cf-4f19-a98f-379bfab0ce70/volumes"
Nov 02 22:10:27.522036 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.522000    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"]
Nov 02 22:10:27.522634 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.522620    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4c450dbd-b4cf-4f19-a98f-379bfab0ce70" containerName="payload-processing"
Nov 02 22:10:27.522684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.522637    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c450dbd-b4cf-4f19-a98f-379bfab0ce70" containerName="payload-processing"
Nov 02 22:10:27.522753 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.522744    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c450dbd-b4cf-4f19-a98f-379bfab0ce70" containerName="payload-processing"
Nov 02 22:10:27.527449 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.527430    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.577995 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.577956    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"]
Nov 02 22:10:27.624903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.624871    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 22:10:27.624903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.624889    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 22:10:27.624903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.624894    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zssxg\""
Nov 02 22:10:27.624903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.624871    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3634f8f9043c\""
Nov 02 22:10:27.625226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.624895    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 22:10:27.625226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.624924    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 22:10:27.656675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.656647    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-plugins-config-volume\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.656831 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.656784    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-routing-overlay-volume\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.656831 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.656827    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmvd\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-kube-api-access-8vmvd\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.656912 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.656870    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-provider-credentials\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.758926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.758884    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-routing-overlay-volume\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.759087 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.758940    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmvd\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-kube-api-access-8vmvd\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.759087 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.758977    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-provider-credentials\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.759087 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.759048    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-plugins-config-volume\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.759725 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.759698    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-routing-overlay-volume\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.759817 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.759800    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-plugins-config-volume\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.761536 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.761513    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-provider-credentials\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.815882 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.815811    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmvd\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-kube-api-access-8vmvd\") pod \"payload-processing-external-model-6f79fb9565-nkqpf\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") " pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:27.841097 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:27.841068    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:28.317199 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:10:28.317167    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca759ac_4ec1_4add_9a99_7a2255f84e84.slice/crio-22bc4a65751bef41d9b6ae66b05b09c4b081624325bdb291d5e96049fcf4ffb7 WatchSource:0}: Error finding container 22bc4a65751bef41d9b6ae66b05b09c4b081624325bdb291d5e96049fcf4ffb7: Status 404 returned error can't find the container with id 22bc4a65751bef41d9b6ae66b05b09c4b081624325bdb291d5e96049fcf4ffb7
Nov 02 22:10:28.319836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:28.319762    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"]
Nov 02 22:10:29.253694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:29.253659    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf" event={"ID":"dca759ac-4ec1-4add-9a99-7a2255f84e84","Type":"ContainerStarted","Data":"385d61d5ed1d5169743ab6409ec76d50e6230d8909ef3d75e30e06870d1a625a"}
Nov 02 22:10:29.254262 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:29.253709    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf" event={"ID":"dca759ac-4ec1-4add-9a99-7a2255f84e84","Type":"ContainerStarted","Data":"22bc4a65751bef41d9b6ae66b05b09c4b081624325bdb291d5e96049fcf4ffb7"}
Nov 02 22:10:29.254262 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:29.253857    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:29.371462 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:29.371404    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf" podStartSLOduration=2.371389551 podStartE2EDuration="2.371389551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:29.367806762 +0000 UTC m=+3716.631343080" watchObservedRunningTime="2026-11-02 22:10:29.371389551 +0000 UTC m=+3716.634925868"
Nov 02 22:10:31.490073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.490030    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"]
Nov 02 22:10:31.494766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.494743    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.540534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.540504    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"]
Nov 02 22:10:31.544673 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.544654    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.590875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.590838    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"]
Nov 02 22:10:31.609013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.608982    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jk4q\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-kube-api-access-2jk4q\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.609196 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609047    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.609271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609190    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.609271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609221    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.609271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609263    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.609518 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609400    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.609518 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609444    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d310373-b874-4397-be84-c81975774f8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.609518 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609480    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.609686 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609520    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.609686 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609591    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.609686 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609617    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d310373-b874-4397-be84-c81975774f8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.609686 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609662    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.609988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609827    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.609988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609859    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.609988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.609889    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe3a4bb-818f-45fd-b27b-688c67d97929-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.610126 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.610012    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.610126 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.610047    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.610126 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.610082    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2z2\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-kube-api-access-ff2z2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.640070 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.640034    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"]
Nov 02 22:10:31.711410 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711371    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.711617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711430    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d310373-b874-4397-be84-c81975774f8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.711617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711463    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.711617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711506    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.711617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711582    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.711617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711608    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d310373-b874-4397-be84-c81975774f8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.711922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711654    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.711922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711703    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.711922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711728    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.711922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711760    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe3a4bb-818f-45fd-b27b-688c67d97929-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.711922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711800    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.711922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711826    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.711922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711841    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.711922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711863    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2z2\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-kube-api-access-ff2z2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.712391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.711932    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jk4q\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-kube-api-access-2jk4q\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.712391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.712001    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.712391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.712055    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.712391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.712089    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.712391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.712168    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.714031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.713049    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.714031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.713079    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.714031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.713341    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.714031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.713889    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.714415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.714322    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.715063 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.714970    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d310373-b874-4397-be84-c81975774f8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.715477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.715333    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.715477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.715436    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.715640 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.715503    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d310373-b874-4397-be84-c81975774f8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.715756 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.715730    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.715906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.715886    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.716466 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.716443    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe3a4bb-818f-45fd-b27b-688c67d97929-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.717083 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.717060    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.813141 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.807148    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2z2\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-kube-api-access-ff2z2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.844434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.840411    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jk4q\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-kube-api-access-2jk4q\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.844434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.841036    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lrnwd\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.844702 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.844678    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hbd6v\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:31.858321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.858289    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:31.862623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:31.862572    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"]
Nov 02 22:10:32.109614 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.109526    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:32.244321 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:10:32.244285    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe3a4bb_818f_45fd_b27b_688c67d97929.slice/crio-e1c091aae3257a1ed288a30fefb2da3b97ac014c71ff258148be7b02908599cf WatchSource:0}: Error finding container e1c091aae3257a1ed288a30fefb2da3b97ac014c71ff258148be7b02908599cf: Status 404 returned error can't find the container with id e1c091aae3257a1ed288a30fefb2da3b97ac014c71ff258148be7b02908599cf
Nov 02 22:10:32.247300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.247269    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"]
Nov 02 22:10:32.247586 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.247551    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:32.247672 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.247631    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:32.247739 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.247674    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:32.277183 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.277090    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" event={"ID":"bfe3a4bb-818f-45fd-b27b-688c67d97929","Type":"ContainerStarted","Data":"e1c091aae3257a1ed288a30fefb2da3b97ac014c71ff258148be7b02908599cf"}
Nov 02 22:10:32.499935 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:10:32.499904    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d310373_b874_4397_be84_c81975774f8d.slice/crio-32e17950443ef6d854d0f37f4fbf55a1922bec3b4a21c59d4382140273a0e72c WatchSource:0}: Error finding container 32e17950443ef6d854d0f37f4fbf55a1922bec3b4a21c59d4382140273a0e72c: Status 404 returned error can't find the container with id 32e17950443ef6d854d0f37f4fbf55a1922bec3b4a21c59d4382140273a0e72c
Nov 02 22:10:32.502684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.502659    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"]
Nov 02 22:10:32.503170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.503140    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:32.503244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.503203    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:32.503244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:32.503236    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:33.286450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:33.286410    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" event={"ID":"2d310373-b874-4397-be84-c81975774f8d","Type":"ContainerStarted","Data":"ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da"}
Nov 02 22:10:33.286665 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:33.286467    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" event={"ID":"2d310373-b874-4397-be84-c81975774f8d","Type":"ContainerStarted","Data":"32e17950443ef6d854d0f37f4fbf55a1922bec3b4a21c59d4382140273a0e72c"}
Nov 02 22:10:33.288183 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:33.288155    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" event={"ID":"bfe3a4bb-818f-45fd-b27b-688c67d97929","Type":"ContainerStarted","Data":"deee69be57f224996c7d7dfd844aa04d97713cb3a76301fe6b6d5b56fb9df6f4"}
Nov 02 22:10:33.426230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:33.423946    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podStartSLOduration=2.423927548 podStartE2EDuration="2.423927548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:33.422604099 +0000 UTC m=+3720.686140430" watchObservedRunningTime="2026-11-02 22:10:33.423927548 +0000 UTC m=+3720.687463866"
Nov 02 22:10:33.566610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:33.566507    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podStartSLOduration=2.566492564 podStartE2EDuration="2.566492564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:33.563203043 +0000 UTC m=+3720.826739361" watchObservedRunningTime="2026-11-02 22:10:33.566492564 +0000 UTC m=+3720.830028954"
Nov 02 22:10:33.859665 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:33.859582    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:33.861287 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:33.861259    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 22:10:33.861416 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:33.861313    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 22:10:34.114195 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:34.111528    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:34.123132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:34.119250    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hbd6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 22:10:34.123132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:34.119317    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 22:10:34.860134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:34.859497    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 22:10:34.860134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:34.859561    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 22:10:35.138192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:35.122687    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hbd6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 22:10:35.138192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:35.122778    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 22:10:35.787345 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:35.787309    4528 scope.go:122] "RemoveContainer" containerID="9ce54752be69d02ce8b0c799fb5ac2ccb87effdd13d5ee0511b520f58c10e138"
Nov 02 22:10:35.800714 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:35.800574    4528 scope.go:122] "RemoveContainer" containerID="465e890fb66fb22f902a82522c010c5bd338fef499de695eb83536c46bbe795d"
Nov 02 22:10:35.859819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:35.859785    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 22:10:35.860139 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:35.860084    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 22:10:36.121193 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:36.116526    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hbd6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 22:10:36.121193 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:36.116590    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 22:10:36.862136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:36.861584    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 22:10:36.862136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:36.861646    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 22:10:36.864314 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:36.864263    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf" podUID="dca759ac-4ec1-4add-9a99-7a2255f84e84" containerName="payload-processing" containerID="cri-o://385d61d5ed1d5169743ab6409ec76d50e6230d8909ef3d75e30e06870d1a625a" gracePeriod=25
Nov 02 22:10:37.126292 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.126205    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hbd6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 22:10:37.127301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.127263    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 22:10:37.455195 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.433344    4528 generic.go:356] "Generic (PLEG): container finished" podID="dca759ac-4ec1-4add-9a99-7a2255f84e84" containerID="385d61d5ed1d5169743ab6409ec76d50e6230d8909ef3d75e30e06870d1a625a" exitCode=0
Nov 02 22:10:37.455617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.455576    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf" event={"ID":"dca759ac-4ec1-4add-9a99-7a2255f84e84","Type":"ContainerDied","Data":"385d61d5ed1d5169743ab6409ec76d50e6230d8909ef3d75e30e06870d1a625a"}
Nov 02 22:10:37.848997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.848962    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:37.873136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.873076    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 22:10:37.873847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.873415    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 22:10:37.956714 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.956256    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-routing-overlay-volume\") pod \"dca759ac-4ec1-4add-9a99-7a2255f84e84\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") "
Nov 02 22:10:37.957020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.956679    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-routing-overlay-volume" pod "dca759ac-4ec1-4add-9a99-7a2255f84e84" (UID: "dca759ac-4ec1-4add-9a99-7a2255f84e84"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:10:37.957234 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.957214    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-provider-credentials\") pod \"dca759ac-4ec1-4add-9a99-7a2255f84e84\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") "
Nov 02 22:10:37.957794 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.957773    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-kube-api-access-8vmvd\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-kube-api-access-8vmvd\") pod \"dca759ac-4ec1-4add-9a99-7a2255f84e84\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") "
Nov 02 22:10:37.958367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.958349    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-plugins-config-volume\") pod \"dca759ac-4ec1-4add-9a99-7a2255f84e84\" (UID: \"dca759ac-4ec1-4add-9a99-7a2255f84e84\") "
Nov 02 22:10:37.959545 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.959521    4528 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-routing-overlay-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:37.975803 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.958908    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-plugins-config-volume" pod "dca759ac-4ec1-4add-9a99-7a2255f84e84" (UID: "dca759ac-4ec1-4add-9a99-7a2255f84e84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:10:37.975973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:37.966580    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-kube-api-access-8vmvd" pod "dca759ac-4ec1-4add-9a99-7a2255f84e84" (UID: "dca759ac-4ec1-4add-9a99-7a2255f84e84"). InnerVolumeSpecName "kube-api-access-8vmvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:10:38.009255 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.009213    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-provider-credentials" pod "dca759ac-4ec1-4add-9a99-7a2255f84e84" (UID: "dca759ac-4ec1-4add-9a99-7a2255f84e84"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:10:38.077254 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.077213    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vmvd\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-kube-api-access-8vmvd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:38.101257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.101177    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca759ac-4ec1-4add-9a99-7a2255f84e84-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:38.101559 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.101535    4528 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dca759ac-4ec1-4add-9a99-7a2255f84e84-provider-credentials\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:38.112054 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.112021    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hbd6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 22:10:38.112332 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.112300    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 22:10:38.662258 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.662216    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf" event={"ID":"dca759ac-4ec1-4add-9a99-7a2255f84e84","Type":"ContainerDied","Data":"22bc4a65751bef41d9b6ae66b05b09c4b081624325bdb291d5e96049fcf4ffb7"}
Nov 02 22:10:38.662903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.662879    4528 scope.go:122] "RemoveContainer" containerID="385d61d5ed1d5169743ab6409ec76d50e6230d8909ef3d75e30e06870d1a625a"
Nov 02 22:10:38.666362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.666337    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"
Nov 02 22:10:38.847287 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.846213    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:10:38.851134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.847931    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:10:38.871341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.871309    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 22:10:38.871575 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:38.871545    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 22:10:39.029252 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.029212    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"]
Nov 02 22:10:39.041272 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.041228    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:10:39.068045 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.068006    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f79fb9565-nkqpf"]
Nov 02 22:10:39.104338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.104290    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:10:39.118151 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.117630    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hbd6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 22:10:39.118151 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.117702    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 22:10:39.144266 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.144215    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:10:39.410392 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.410279    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Nov 02 22:10:39.421714 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.421677    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"]
Nov 02 22:10:39.422976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.422951    4528 cpu_manager.go:414] "containerMap: removing container" podUID="dca759ac-4ec1-4add-9a99-7a2255f84e84" containerName="payload-processing"
Nov 02 22:10:39.423153 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.423138    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca759ac-4ec1-4add-9a99-7a2255f84e84" containerName="payload-processing"
Nov 02 22:10:39.423491 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.423476    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca759ac-4ec1-4add-9a99-7a2255f84e84" containerName="payload-processing"
Nov 02 22:10:39.437755 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.429581    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.484546 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.484511    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 22:10:39.495140 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.485546    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dxclp\""
Nov 02 22:10:39.495561 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.495532    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9cc885af3cb5\""
Nov 02 22:10:39.499137 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.495811    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 22:10:39.507129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.499383    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 22:10:39.563391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.563357    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"]
Nov 02 22:10:39.585401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.585370    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 22:10:39.661459 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.661356    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.661659 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.661491    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k9cv\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-kube-api-access-6k9cv\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.661759 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.661657    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-routing-overlay-volume\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.661759 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.661700    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.763386 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.763346    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-routing-overlay-volume\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.763857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.763835    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.763969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.763916    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.764047 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.764032    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k9cv\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-kube-api-access-6k9cv\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.764242 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.764215    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-routing-overlay-volume\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.764823 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.764792    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.766836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.766804    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.840950 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.840907    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k9cv\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-kube-api-access-6k9cv\") pod \"payload-processing-external-model-69f4b58446-qlss4\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") " pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:39.859268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.859235    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 22:10:39.859453 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.859310    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 22:10:39.871779 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:39.871746    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:40.140049 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.139234    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hbd6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 22:10:40.140049 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.139295    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 22:10:40.468151 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.463683    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"]
Nov 02 22:10:40.693651 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.693609    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4" event={"ID":"10bedcff-5437-4e2f-87ba-1c55ea2e22d8","Type":"ContainerStarted","Data":"74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262"}
Nov 02 22:10:40.693828 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.693679    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4" event={"ID":"10bedcff-5437-4e2f-87ba-1c55ea2e22d8","Type":"ContainerStarted","Data":"ef4af9228f5eac2412b6d967060cf0d424f846faaaffc4eeffff436cdc9eff6c"}
Nov 02 22:10:40.694708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.694686    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:40.786544 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:40.786493    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9cc885af3cb5: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:40.787026 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:40.787007    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69f4b58446-qlss4: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:40.787215 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:40.787203    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials podName:10bedcff-5437-4e2f-87ba-1c55ea2e22d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:41.287180128 +0000 UTC m=+3728.550716429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials") pod "payload-processing-external-model-69f4b58446-qlss4" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8") : secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:40.821685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.821613    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4" podStartSLOduration=1.821593011 podStartE2EDuration="1.821593011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:40.819644849 +0000 UTC m=+3728.083181165" watchObservedRunningTime="2026-11-02 22:10:40.821593011 +0000 UTC m=+3728.085129330"
Nov 02 22:10:40.859524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.859496    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 22:10:40.859753 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.859731    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 22:10:40.968995 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:40.960749    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca759ac-4ec1-4add-9a99-7a2255f84e84" path="/var/lib/kubelet/pods/dca759ac-4ec1-4add-9a99-7a2255f84e84/volumes"
Nov 02 22:10:41.117134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:41.116511    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hbd6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 22:10:41.117134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:41.116574    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 22:10:41.293951 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:41.293910    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9cc885af3cb5: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:41.293951 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:41.293948    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69f4b58446-qlss4: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:41.294215 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:41.294019    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials podName:10bedcff-5437-4e2f-87ba-1c55ea2e22d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:42.293997836 +0000 UTC m=+3729.557534155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials") pod "payload-processing-external-model-69f4b58446-qlss4" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8") : secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:42.093248 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:42.093208    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"]
Nov 02 22:10:42.368274 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:42.368182    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9cc885af3cb5: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:42.368526 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:42.368510    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69f4b58446-qlss4: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:42.368711 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:42.368699    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials podName:10bedcff-5437-4e2f-87ba-1c55ea2e22d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:44.368676412 +0000 UTC m=+3731.632212727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials") pod "payload-processing-external-model-69f4b58446-qlss4" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8") : secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:42.789747 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:42.789711    4528 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4" secret="" err="secret \"payload-processing-external-model-dockercfg-dxclp\" not found"
Nov 02 22:10:42.892000 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:42.891956    4528 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 22:10:42.892510 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:42.892491    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume podName:10bedcff-5437-4e2f-87ba-1c55ea2e22d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:43.392469093 +0000 UTC m=+3730.656005404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume") pod "payload-processing-external-model-69f4b58446-qlss4" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8") : configmap "payload-processing-external-model-plugins" not found
Nov 02 22:10:42.903368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:42.903320    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:10:42.903640 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:42.903598    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:10:43.159576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:43.159470    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hbd6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:10:43.159576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:43.159541    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:10:43.442189 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:43.439230    4528 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 22:10:43.442189 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:43.439318    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume podName:10bedcff-5437-4e2f-87ba-1c55ea2e22d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:44.43929824 +0000 UTC m=+3731.702834556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume") pod "payload-processing-external-model-69f4b58446-qlss4" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8") : configmap "payload-processing-external-model-plugins" not found
Nov 02 22:10:43.446139 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:43.445876    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:43.446826 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:43.446597    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:43.448493 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:43.448472    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:10:43.568736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:43.568698    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:43.569220 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:43.569169    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:43.570689 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:43.570662    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:10:44.136205 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:44.135227    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:10:44.465215 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:44.464608    4528 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 22:10:44.465215 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:44.464695    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume podName:10bedcff-5437-4e2f-87ba-1c55ea2e22d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:46.464673411 +0000 UTC m=+3733.728209707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume") pod "payload-processing-external-model-69f4b58446-qlss4" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8") : configmap "payload-processing-external-model-plugins" not found
Nov 02 22:10:44.465215 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:44.464812    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9cc885af3cb5: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:44.465215 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:44.464837    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69f4b58446-qlss4: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:44.465215 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:44.464889    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials podName:10bedcff-5437-4e2f-87ba-1c55ea2e22d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:48.464871891 +0000 UTC m=+3735.728408204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials") pod "payload-processing-external-model-69f4b58446-qlss4" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8") : secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:44.532160 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.532095    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"]
Nov 02 22:10:44.538220 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.538191    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.582308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.582271    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7clzm\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-kube-api-access-7clzm\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.583039 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.582738    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-socket\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.583221 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.583202    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-certs\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.583416 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.583399    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-envoy\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.583679 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.583659    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-podinfo\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.583884 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.583863    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-token\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.634352 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.634316    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istiod-ca-cert\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.634686 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.634659    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-credential-socket\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.634860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.634841    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-data\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.635094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.611317    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-065be7-openshift-default-dockercfg-bzkhn\""
Nov 02 22:10:44.636298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.636255    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"]
Nov 02 22:10:44.739192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739017    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-socket\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.739192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739068    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-certs\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.739192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739124    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-envoy\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.739504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739261    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-podinfo\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.739504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739291    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-token\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.739504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739325    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istiod-ca-cert\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.739504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739376    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-credential-socket\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.739504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739410    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-data\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.739504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739493    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7clzm\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-kube-api-access-7clzm\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.749698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.739937    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-socket\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.774511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.759867    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istiod-ca-cert\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.774511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.760529    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-credential-socket\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.774511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.760667    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-data\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.774511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.760786    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-certs\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.789148 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.789050    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-podinfo\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.791514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.791455    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-envoy\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.807851 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.807793    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7clzm\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-kube-api-access-7clzm\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.818709 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.818629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-token\") pod \"e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") " pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:44.877201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:44.876916    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:45.374291 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:45.371911    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:45.374291 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:45.372002    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:45.374291 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:45.372046    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:45.378374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:45.378351    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"]
Nov 02 22:10:45.927133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:45.909389    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" event={"ID":"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70","Type":"ContainerStarted","Data":"b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b"}
Nov 02 22:10:45.927133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:45.909463    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" event={"ID":"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70","Type":"ContainerStarted","Data":"ba045456a9bb4c47636956e36ab3f83a30eb67c00287a1876bae7f821615a33b"}
Nov 02 22:10:46.083135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:46.082708    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" podStartSLOduration=2.082693862 podStartE2EDuration="2.082693862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:46.080522124 +0000 UTC m=+3733.344058457" watchObservedRunningTime="2026-11-02 22:10:46.082693862 +0000 UTC m=+3733.346230179"
Nov 02 22:10:46.536130 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:46.535850    4528 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 22:10:46.536130 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:46.535934    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume podName:10bedcff-5437-4e2f-87ba-1c55ea2e22d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:50.53591391 +0000 UTC m=+3737.799450229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume") pod "payload-processing-external-model-69f4b58446-qlss4" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8") : configmap "payload-processing-external-model-plugins" not found
Nov 02 22:10:46.878926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:46.878834    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:46.886534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:46.886503    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:46.918153 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:46.918093    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:46.919079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:46.919052    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:10:48.477397 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:48.477364    4528 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9cc885af3cb5: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:48.477397 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:48.477394    4528 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69f4b58446-qlss4: secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:48.477941 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:48.477460    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials podName:10bedcff-5437-4e2f-87ba-1c55ea2e22d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:56.477439168 +0000 UTC m=+3743.740975464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials") pod "payload-processing-external-model-69f4b58446-qlss4" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8") : secret "agc-cred-9cc885af3cb5" not found
Nov 02 22:10:48.818010 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:48.817965    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4" podUID="10bedcff-5437-4e2f-87ba-1c55ea2e22d8" containerName="payload-processing" containerID="cri-o://74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262" gracePeriod=25
Nov 02 22:10:48.933609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:48.933572    4528 generic.go:356] "Generic (PLEG): container finished" podID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerID="cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96" exitCode=0
Nov 02 22:10:48.934064 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:48.933670    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" event={"ID":"bfd786b7-e883-43a6-b539-c7c152512c0a","Type":"ContainerDied","Data":"cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96"}
Nov 02 22:10:48.934415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:48.934398    4528 scope.go:122] "RemoveContainer" containerID="cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96"
Nov 02 22:10:48.935904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:48.935874    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:48.936000 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:48.935949    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:48.936000 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:48.935984    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:10:49.156953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.156910    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:49.298528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.298489    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume\") pod \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") "
Nov 02 22:10:49.298739 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.298533    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials\") pod \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") "
Nov 02 22:10:49.298739 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.298566    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-routing-overlay-volume\") pod \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") "
Nov 02 22:10:49.298739 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.298589    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-kube-api-access-6k9cv\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-kube-api-access-6k9cv\") pod \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\" (UID: \"10bedcff-5437-4e2f-87ba-1c55ea2e22d8\") "
Nov 02 22:10:49.299017 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.298984    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-routing-overlay-volume" pod "10bedcff-5437-4e2f-87ba-1c55ea2e22d8" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:10:49.299089 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.299030    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume" pod "10bedcff-5437-4e2f-87ba-1c55ea2e22d8" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:10:49.299215 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.299184    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:49.299281 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.299213    4528 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-routing-overlay-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:49.301408 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.301337    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-kube-api-access-6k9cv" pod "10bedcff-5437-4e2f-87ba-1c55ea2e22d8" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8"). InnerVolumeSpecName "kube-api-access-6k9cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:10:49.302026 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.301998    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials" pod "10bedcff-5437-4e2f-87ba-1c55ea2e22d8" (UID: "10bedcff-5437-4e2f-87ba-1c55ea2e22d8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:10:49.400273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.400188    4528 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-provider-credentials\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:49.400273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.400218    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k9cv\" (UniqueName: \"kubernetes.io/projected/10bedcff-5437-4e2f-87ba-1c55ea2e22d8-kube-api-access-6k9cv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:10:49.942864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.942825    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" event={"ID":"bfd786b7-e883-43a6-b539-c7c152512c0a","Type":"ContainerStarted","Data":"b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e"}
Nov 02 22:10:49.944394 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.944363    4528 generic.go:356] "Generic (PLEG): container finished" podID="10bedcff-5437-4e2f-87ba-1c55ea2e22d8" containerID="74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262" exitCode=0
Nov 02 22:10:49.944530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.944408    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"
Nov 02 22:10:49.944530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.944445    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4" event={"ID":"10bedcff-5437-4e2f-87ba-1c55ea2e22d8","Type":"ContainerDied","Data":"74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262"}
Nov 02 22:10:49.944530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.944498    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69f4b58446-qlss4" event={"ID":"10bedcff-5437-4e2f-87ba-1c55ea2e22d8","Type":"ContainerDied","Data":"ef4af9228f5eac2412b6d967060cf0d424f846faaaffc4eeffff436cdc9eff6c"}
Nov 02 22:10:49.944530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.944526    4528 scope.go:122] "RemoveContainer" containerID="74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262"
Nov 02 22:10:49.955275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.955261    4528 scope.go:122] "RemoveContainer" containerID="74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262"
Nov 02 22:10:49.955558 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:10:49.955536    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262\": container with ID starting with 74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262 not found: ID does not exist" containerID="74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262"
Nov 02 22:10:49.955648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:49.955564    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262"} err="failed to get container status \"74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262\": rpc error: code = NotFound desc = could not find container \"74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262\": container with ID starting with 74d9b42620a1c87d8bfdde0f34ab83616c3241f2f084138449c9d898a00bc262 not found: ID does not exist"
Nov 02 22:10:50.178855 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:50.178816    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"]
Nov 02 22:10:50.182092 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:50.182064    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-69f4b58446-qlss4"]
Nov 02 22:10:50.629441 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:50.629406    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:10:50.630817 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:50.630791    4528 patch_prober.go:29] interesting pod/e2e-autores-363e15-openshift-default-754479d777-k44vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 02 22:10:50.630977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:50.630839    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 02 22:10:50.933447 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:50.933355    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bedcff-5437-4e2f-87ba-1c55ea2e22d8" path="/var/lib/kubelet/pods/10bedcff-5437-4e2f-87ba-1c55ea2e22d8/volumes"
Nov 02 22:10:51.630586 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:51.630546    4528 patch_prober.go:29] interesting pod/e2e-autores-363e15-openshift-default-754479d777-k44vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 02 22:10:51.631007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:51.630615    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 02 22:10:52.630432 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:52.630393    4528 patch_prober.go:29] interesting pod/e2e-autores-363e15-openshift-default-754479d777-k44vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 02 22:10:52.630637 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:52.630449    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 02 22:10:53.274894 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.274811    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"]
Nov 02 22:10:53.275859 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.275820    4528 cpu_manager.go:414] "containerMap: removing container" podUID="10bedcff-5437-4e2f-87ba-1c55ea2e22d8" containerName="payload-processing"
Nov 02 22:10:53.275859 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.275859    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bedcff-5437-4e2f-87ba-1c55ea2e22d8" containerName="payload-processing"
Nov 02 22:10:53.276143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.276014    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bedcff-5437-4e2f-87ba-1c55ea2e22d8" containerName="payload-processing"
Nov 02 22:10:53.281776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.281752    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:10:53.295826 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.295800    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:10:53.337289 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.335463    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"]
Nov 02 22:10:53.360132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.358420    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-065be7-dockercfg-tr6fj\""
Nov 02 22:10:53.360132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.358516    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-065be7\""
Nov 02 22:10:53.432132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.431800    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ae8f91-cece-4328-a38f-fc8cdb44607a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx\" (UID: \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:10:53.432132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.431866    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jns2g\" (UniqueName: \"kubernetes.io/projected/a9ae8f91-cece-4328-a38f-fc8cdb44607a-kube-api-access-jns2g\") pod \"payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx\" (UID: \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:10:53.523991 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.522474    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"]
Nov 02 22:10:53.570032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.567646    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jns2g\" (UniqueName: \"kubernetes.io/projected/a9ae8f91-cece-4328-a38f-fc8cdb44607a-kube-api-access-jns2g\") pod \"payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx\" (UID: \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:10:53.570032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.567923    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ae8f91-cece-4328-a38f-fc8cdb44607a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx\" (UID: \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:10:53.570032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.568950    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ae8f91-cece-4328-a38f-fc8cdb44607a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx\" (UID: \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:10:53.571595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.571573    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:10:53.573120 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.573080    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"]
Nov 02 22:10:53.630424 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.630381    4528 patch_prober.go:29] interesting pod/e2e-autores-363e15-openshift-default-754479d777-k44vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 02 22:10:53.630631 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.630444    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 02 22:10:53.635035 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.635007    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jns2g\" (UniqueName: \"kubernetes.io/projected/a9ae8f91-cece-4328-a38f-fc8cdb44607a-kube-api-access-jns2g\") pod \"payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx\" (UID: \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:10:53.668594 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.668563    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f72523-e582-4041-aa7b-159b17143c40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj\" (UID: \"f8f72523-e582-4041-aa7b-159b17143c40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:10:53.668763 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.668693    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffdm8\" (UniqueName: \"kubernetes.io/projected/f8f72523-e582-4041-aa7b-159b17143c40-kube-api-access-ffdm8\") pod \"payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj\" (UID: \"f8f72523-e582-4041-aa7b-159b17143c40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:10:53.770380 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.770341    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f72523-e582-4041-aa7b-159b17143c40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj\" (UID: \"f8f72523-e582-4041-aa7b-159b17143c40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:10:53.770541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.770481    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffdm8\" (UniqueName: \"kubernetes.io/projected/f8f72523-e582-4041-aa7b-159b17143c40-kube-api-access-ffdm8\") pod \"payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj\" (UID: \"f8f72523-e582-4041-aa7b-159b17143c40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:10:53.771048 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.771026    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f72523-e582-4041-aa7b-159b17143c40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj\" (UID: \"f8f72523-e582-4041-aa7b-159b17143c40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:10:53.829664 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.829588    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffdm8\" (UniqueName: \"kubernetes.io/projected/f8f72523-e582-4041-aa7b-159b17143c40-kube-api-access-ffdm8\") pod \"payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj\" (UID: \"f8f72523-e582-4041-aa7b-159b17143c40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:10:53.891445 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.891404    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:10:53.896771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:53.896748    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:10:54.292053 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:10:54.292023    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f72523_e582_4041_aa7b_159b17143c40.slice/crio-9661ceb492c01c1dcf668a9b434838aee4fb0143e0631d65e19b4a06321a11f9 WatchSource:0}: Error finding container 9661ceb492c01c1dcf668a9b434838aee4fb0143e0631d65e19b4a06321a11f9: Status 404 returned error can't find the container with id 9661ceb492c01c1dcf668a9b434838aee4fb0143e0631d65e19b4a06321a11f9
Nov 02 22:10:54.294845 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:54.294820    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"]
Nov 02 22:10:54.338868 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:10:54.338833    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ae8f91_cece_4328_a38f_fc8cdb44607a.slice/crio-d84bb1d7905b3f3564d93647c17a50861052e4ff606124791c3ab20893eb6ed1 WatchSource:0}: Error finding container d84bb1d7905b3f3564d93647c17a50861052e4ff606124791c3ab20893eb6ed1: Status 404 returned error can't find the container with id d84bb1d7905b3f3564d93647c17a50861052e4ff606124791c3ab20893eb6ed1
Nov 02 22:10:54.342285 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:54.342169    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"]
Nov 02 22:10:54.630483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:54.630393    4528 patch_prober.go:29] interesting pod/e2e-autores-363e15-openshift-default-754479d777-k44vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 02 22:10:54.630483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:54.630448    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 02 22:10:55.086338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.086300    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj" event={"ID":"f8f72523-e582-4041-aa7b-159b17143c40","Type":"ContainerStarted","Data":"b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253"}
Nov 02 22:10:55.086884 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.086855    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj" event={"ID":"f8f72523-e582-4041-aa7b-159b17143c40","Type":"ContainerStarted","Data":"9661ceb492c01c1dcf668a9b434838aee4fb0143e0631d65e19b4a06321a11f9"}
Nov 02 22:10:55.087011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.086894    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:10:55.088159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.088133    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx" event={"ID":"a9ae8f91-cece-4328-a38f-fc8cdb44607a","Type":"ContainerStarted","Data":"d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4"}
Nov 02 22:10:55.088251 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.088177    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx" event={"ID":"a9ae8f91-cece-4328-a38f-fc8cdb44607a","Type":"ContainerStarted","Data":"d84bb1d7905b3f3564d93647c17a50861052e4ff606124791c3ab20893eb6ed1"}
Nov 02 22:10:55.088251 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.088203    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:10:55.214134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.212840    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj" podStartSLOduration=2.212820283 podStartE2EDuration="2.212820283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:55.199990858 +0000 UTC m=+3742.463527198" watchObservedRunningTime="2026-11-02 22:10:55.212820283 +0000 UTC m=+3742.476356602"
Nov 02 22:10:55.337712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.335480    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx" podStartSLOduration=2.335457685 podStartE2EDuration="2.335457685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:55.328391666 +0000 UTC m=+3742.591927997" watchObservedRunningTime="2026-11-02 22:10:55.335457685 +0000 UTC m=+3742.598994004"
Nov 02 22:10:55.634371 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.634285    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:10:55.635382 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:10:55.635361    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:11:06.108101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:06.104778    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:11:07.105301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:07.105265    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:11:14.117386 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:11:14.117341    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:11:21.291839 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:21.291802    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"]
Nov 02 22:11:21.389038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:21.388992    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"]
Nov 02 22:11:22.866397 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:22.866357    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:11:26.292782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.292730    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d" podUID="4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc" containerName="payload-processing" containerID="cri-o://0d818287dde90c2210e0405ea76a692dd2cb30339f70056d2fa7d0b4aeee1c14" gracePeriod=25
Nov 02 22:11:26.361263 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.361227    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"]
Nov 02 22:11:26.389653 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.389607    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf" podUID="cb5c2d52-04db-4583-baa8-9afd8a37dec1" containerName="payload-pre-processing" containerID="cri-o://1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218" gracePeriod=25
Nov 02 22:11:26.411977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.411948    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9981d2-nf464"]
Nov 02 22:11:26.440730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.440697    4528 generic.go:356] "Generic (PLEG): container finished" podID="4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc" containerID="0d818287dde90c2210e0405ea76a692dd2cb30339f70056d2fa7d0b4aeee1c14" exitCode=0
Nov 02 22:11:26.440913 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.440770    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d" event={"ID":"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc","Type":"ContainerDied","Data":"0d818287dde90c2210e0405ea76a692dd2cb30339f70056d2fa7d0b4aeee1c14"}
Nov 02 22:11:26.604816 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.604766    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"]
Nov 02 22:11:26.607168 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.605465    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy" containerID="cri-o://96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d" gracePeriod=30
Nov 02 22:11:26.918945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.918908    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99d66ec-4b6c-4c38-b3a3-34907e57185a" path="/var/lib/kubelet/pods/e99d66ec-4b6c-4c38-b3a3-34907e57185a/volumes"
Nov 02 22:11:26.938722 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.938687    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:11:26.960528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:26.960503    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:11:27.079550 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.079499    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-plugins-config-volume\") pod \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\" (UID: \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\") "
Nov 02 22:11:27.079757 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.079624    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5c2d52-04db-4583-baa8-9afd8a37dec1-kube-api-access-l5sbv\" (UniqueName: \"kubernetes.io/projected/cb5c2d52-04db-4583-baa8-9afd8a37dec1-kube-api-access-l5sbv\") pod \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\" (UID: \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\") "
Nov 02 22:11:27.079757 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.079743    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb5c2d52-04db-4583-baa8-9afd8a37dec1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5c2d52-04db-4583-baa8-9afd8a37dec1-plugins-config-volume\") pod \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\" (UID: \"cb5c2d52-04db-4583-baa8-9afd8a37dec1\") "
Nov 02 22:11:27.079879 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.079823    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-plugins-config-volume" pod "4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc" (UID: "4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:11:27.079968 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.079952    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-kube-api-access-dqds6\" (UniqueName: \"kubernetes.io/projected/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-kube-api-access-dqds6\") pod \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\" (UID: \"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc\") "
Nov 02 22:11:27.080246 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.080221    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5c2d52-04db-4583-baa8-9afd8a37dec1-plugins-config-volume" pod "cb5c2d52-04db-4583-baa8-9afd8a37dec1" (UID: "cb5c2d52-04db-4583-baa8-9afd8a37dec1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:11:27.080931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.080875    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:27.080931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.080913    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5c2d52-04db-4583-baa8-9afd8a37dec1-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:27.082101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.082071    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5c2d52-04db-4583-baa8-9afd8a37dec1-kube-api-access-l5sbv" pod "cb5c2d52-04db-4583-baa8-9afd8a37dec1" (UID: "cb5c2d52-04db-4583-baa8-9afd8a37dec1"). InnerVolumeSpecName "kube-api-access-l5sbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:11:27.082321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.082295    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-kube-api-access-dqds6" pod "4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc" (UID: "4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc"). InnerVolumeSpecName "kube-api-access-dqds6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:11:27.181994 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.181903    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqds6\" (UniqueName: \"kubernetes.io/projected/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc-kube-api-access-dqds6\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:27.181994 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.181937    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5sbv\" (UniqueName: \"kubernetes.io/projected/cb5c2d52-04db-4583-baa8-9afd8a37dec1-kube-api-access-l5sbv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:27.449342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.449246    4528 generic.go:356] "Generic (PLEG): container finished" podID="cb5c2d52-04db-4583-baa8-9afd8a37dec1" containerID="1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218" exitCode=0
Nov 02 22:11:27.449342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.449312    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"
Nov 02 22:11:27.449342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.449324    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf" event={"ID":"cb5c2d52-04db-4583-baa8-9afd8a37dec1","Type":"ContainerDied","Data":"1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218"}
Nov 02 22:11:27.449943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.449415    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf" event={"ID":"cb5c2d52-04db-4583-baa8-9afd8a37dec1","Type":"ContainerDied","Data":"0646ff25e85bfe05cc2b1bd29a34f64b1b8c0f2bdcdcadb411b0ae98cb705f4e"}
Nov 02 22:11:27.449943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.449449    4528 scope.go:122] "RemoveContainer" containerID="1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218"
Nov 02 22:11:27.451086 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.451067    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"
Nov 02 22:11:27.451233 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.451078    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d" event={"ID":"4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc","Type":"ContainerDied","Data":"abd9a5bf12034a784aa70cae8b2ce3463e8a567661a5b31ac95e4dc8c036c031"}
Nov 02 22:11:27.461283 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.461219    4528 scope.go:122] "RemoveContainer" containerID="1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218"
Nov 02 22:11:27.461641 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:11:27.461603    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218\": container with ID starting with 1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218 not found: ID does not exist" containerID="1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218"
Nov 02 22:11:27.461713 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.461652    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218"} err="failed to get container status \"1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218\": rpc error: code = NotFound desc = could not find container \"1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218\": container with ID starting with 1bb0006559bb23202679c23895e88becef04381504415146838d708d2e934218 not found: ID does not exist"
Nov 02 22:11:27.461713 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.461673    4528 scope.go:122] "RemoveContainer" containerID="0d818287dde90c2210e0405ea76a692dd2cb30339f70056d2fa7d0b4aeee1c14"
Nov 02 22:11:27.620012 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.619975    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"]
Nov 02 22:11:27.667402 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.667366    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6ba06f-5b4bb9c79-zffgf"]
Nov 02 22:11:27.734161 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.734040    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"]
Nov 02 22:11:27.738823 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.738787    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6ba06f-849675b7-xk25d"]
Nov 02 22:11:27.744173 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.744101    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"]
Nov 02 22:11:27.747118 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:27.747088    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6689da-g8ng8"]
Nov 02 22:11:28.137338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:28.137302    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:11:28.137650 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:28.137619    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:11:28.914930 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:28.914897    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc" path="/var/lib/kubelet/pods/4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc/volumes"
Nov 02 22:11:28.915372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:28.915339    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5c2d52-04db-4583-baa8-9afd8a37dec1" path="/var/lib/kubelet/pods/cb5c2d52-04db-4583-baa8-9afd8a37dec1/volumes"
Nov 02 22:11:28.915734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:28.915720    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc02305c-98ad-4d66-b8d0-8955d6d251c7" path="/var/lib/kubelet/pods/fc02305c-98ad-4d66-b8d0-8955d6d251c7/volumes"
Nov 02 22:11:31.900666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:31.900606    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:11:32.081370 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081336    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-credential-socket\") pod \"0d56d731-5280-4acc-bcda-a5eba4540f36\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") "
Nov 02 22:11:32.081562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081398    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-envoy\") pod \"0d56d731-5280-4acc-bcda-a5eba4540f36\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") "
Nov 02 22:11:32.081562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081428    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-certs\") pod \"0d56d731-5280-4acc-bcda-a5eba4540f36\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") "
Nov 02 22:11:32.081562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081450    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-token\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-token\") pod \"0d56d731-5280-4acc-bcda-a5eba4540f36\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") "
Nov 02 22:11:32.081562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081470    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-podinfo\") pod \"0d56d731-5280-4acc-bcda-a5eba4540f36\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") "
Nov 02 22:11:32.081562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081537    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-kube-api-access-qfg7w\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-kube-api-access-qfg7w\") pod \"0d56d731-5280-4acc-bcda-a5eba4540f36\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") "
Nov 02 22:11:32.081861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081620    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d56d731-5280-4acc-bcda-a5eba4540f36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56d731-5280-4acc-bcda-a5eba4540f36-istiod-ca-cert\") pod \"0d56d731-5280-4acc-bcda-a5eba4540f36\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") "
Nov 02 22:11:32.081861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081671    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-credential-socket" pod "0d56d731-5280-4acc-bcda-a5eba4540f36" (UID: "0d56d731-5280-4acc-bcda-a5eba4540f36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:11:32.081861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081683    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-data\") pod \"0d56d731-5280-4acc-bcda-a5eba4540f36\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") "
Nov 02 22:11:32.081861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081776    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-certs" pod "0d56d731-5280-4acc-bcda-a5eba4540f36" (UID: "0d56d731-5280-4acc-bcda-a5eba4540f36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:11:32.081861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081778    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-socket\") pod \"0d56d731-5280-4acc-bcda-a5eba4540f36\" (UID: \"0d56d731-5280-4acc-bcda-a5eba4540f36\") "
Nov 02 22:11:32.081861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081861    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-data" pod "0d56d731-5280-4acc-bcda-a5eba4540f36" (UID: "0d56d731-5280-4acc-bcda-a5eba4540f36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:11:32.082234 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.081967    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-socket" pod "0d56d731-5280-4acc-bcda-a5eba4540f36" (UID: "0d56d731-5280-4acc-bcda-a5eba4540f36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:11:32.082301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.082264    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d56d731-5280-4acc-bcda-a5eba4540f36-istiod-ca-cert" pod "0d56d731-5280-4acc-bcda-a5eba4540f36" (UID: "0d56d731-5280-4acc-bcda-a5eba4540f36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:11:32.082568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.082546    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:32.082652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.082571    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:32.082652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.082586    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:32.082652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.082599    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:32.082652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.082612    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56d731-5280-4acc-bcda-a5eba4540f36-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:32.083852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.083824    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-kube-api-access-qfg7w" pod "0d56d731-5280-4acc-bcda-a5eba4540f36" (UID: "0d56d731-5280-4acc-bcda-a5eba4540f36"). InnerVolumeSpecName "kube-api-access-qfg7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:11:32.083852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.083835    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-podinfo" pod "0d56d731-5280-4acc-bcda-a5eba4540f36" (UID: "0d56d731-5280-4acc-bcda-a5eba4540f36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:11:32.084102 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.084085    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-token" pod "0d56d731-5280-4acc-bcda-a5eba4540f36" (UID: "0d56d731-5280-4acc-bcda-a5eba4540f36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:11:32.084207 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.084133    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-envoy" pod "0d56d731-5280-4acc-bcda-a5eba4540f36" (UID: "0d56d731-5280-4acc-bcda-a5eba4540f36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:11:32.184321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.184239    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:32.184321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.184265    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:32.184321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.184275    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56d731-5280-4acc-bcda-a5eba4540f36-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:32.184321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.184285    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfg7w\" (UniqueName: \"kubernetes.io/projected/0d56d731-5280-4acc-bcda-a5eba4540f36-kube-api-access-qfg7w\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:32.499374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.499281    4528 generic.go:356] "Generic (PLEG): container finished" podID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerID="96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d" exitCode=0
Nov 02 22:11:32.499374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.499347    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"
Nov 02 22:11:32.499374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.499363    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" event={"ID":"0d56d731-5280-4acc-bcda-a5eba4540f36","Type":"ContainerDied","Data":"96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d"}
Nov 02 22:11:32.499607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.499447    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj" event={"ID":"0d56d731-5280-4acc-bcda-a5eba4540f36","Type":"ContainerDied","Data":"28b07a821d3c7c1d4d3dc48b091c6af63cfc551c67ac0b7cf5e9ac5678a1b3ea"}
Nov 02 22:11:32.499607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.499470    4528 scope.go:122] "RemoveContainer" containerID="96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d"
Nov 02 22:11:32.510546 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.510523    4528 scope.go:122] "RemoveContainer" containerID="3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f"
Nov 02 22:11:32.519830 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.519810    4528 scope.go:122] "RemoveContainer" containerID="96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d"
Nov 02 22:11:32.520074 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:11:32.520054    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d\": container with ID starting with 96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d not found: ID does not exist" containerID="96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d"
Nov 02 22:11:32.520175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.520084    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d"} err="failed to get container status \"96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d\": rpc error: code = NotFound desc = could not find container \"96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d\": container with ID starting with 96a1ed531d8eab39d06ec5aac37bf6c1cf8f7fc605e68b8ff506ba6a501d0d9d not found: ID does not exist"
Nov 02 22:11:32.520175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.520101    4528 scope.go:122] "RemoveContainer" containerID="3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f"
Nov 02 22:11:32.520321 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:11:32.520307    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f\": container with ID starting with 3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f not found: ID does not exist" containerID="3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f"
Nov 02 22:11:32.520367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.520323    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f"} err="failed to get container status \"3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f\": rpc error: code = NotFound desc = could not find container \"3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f\": container with ID starting with 3f8260bcd108cd341d02757257bb1b3cf4c19b5600ee3331bd7a07a98f48b98f not found: ID does not exist"
Nov 02 22:11:32.665129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.665086    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"]
Nov 02 22:11:32.715687 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.715653    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9981d2-openshift-default-5b6875464f-hqqtj"]
Nov 02 22:11:32.918640 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:32.918603    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" path="/var/lib/kubelet/pods/0d56d731-5280-4acc-bcda-a5eba4540f36/volumes"
Nov 02 22:11:35.973780 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:35.973748    4528 scope.go:122] "RemoveContainer" containerID="4819f50dea9e84a14e48ad5d55b7919334524b44be02c818a702a1a2f908d40d"
Nov 02 22:11:35.983468 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:35.983447    4528 scope.go:122] "RemoveContainer" containerID="280bfdedbced75a46e50e51e952bc226af4e86f536c43119e4474b187e90402d"
Nov 02 22:11:36.286914 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:36.286877    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"]
Nov 02 22:11:36.392908 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:36.392870    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"]
Nov 02 22:11:40.224901 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.224873    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"]
Nov 02 22:11:40.225543 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225521    4528 cpu_manager.go:414] "containerMap: removing container" podUID="cb5c2d52-04db-4583-baa8-9afd8a37dec1" containerName="payload-pre-processing"
Nov 02 22:11:40.225543 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225540    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5c2d52-04db-4583-baa8-9afd8a37dec1" containerName="payload-pre-processing"
Nov 02 22:11:40.225698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225557    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy"
Nov 02 22:11:40.225698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225564    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy"
Nov 02 22:11:40.225698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225597    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy"
Nov 02 22:11:40.225698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225604    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy"
Nov 02 22:11:40.225698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225635    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc" containerName="payload-processing"
Nov 02 22:11:40.225698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225641    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc" containerName="payload-processing"
Nov 02 22:11:40.225899 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225736    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy"
Nov 02 22:11:40.225899 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225753    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee7d18c-1ab7-40da-a3f3-9ad82080d8bc" containerName="payload-processing"
Nov 02 22:11:40.225899 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.225763    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5c2d52-04db-4583-baa8-9afd8a37dec1" containerName="payload-pre-processing"
Nov 02 22:11:40.229684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.229664    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:40.281020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.280989    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"]
Nov 02 22:11:40.282262 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.282235    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4585ee-dockercfg-gmq7v\""
Nov 02 22:11:40.282716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.282308    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4585ee\""
Nov 02 22:11:40.298436 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.298394    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3014cf-9f59-4bd5-9d00-d8963e680bee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m\" (UID: \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:40.298958 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.298475    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtcxd\" (UniqueName: \"kubernetes.io/projected/1b3014cf-9f59-4bd5-9d00-d8963e680bee-kube-api-access-wtcxd\") pod \"payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m\" (UID: \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:40.415928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.415890    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3014cf-9f59-4bd5-9d00-d8963e680bee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m\" (UID: \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:40.416241 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.416213    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtcxd\" (UniqueName: \"kubernetes.io/projected/1b3014cf-9f59-4bd5-9d00-d8963e680bee-kube-api-access-wtcxd\") pod \"payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m\" (UID: \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:40.430773 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.430739    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3014cf-9f59-4bd5-9d00-d8963e680bee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m\" (UID: \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:40.449878 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.449842    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"]
Nov 02 22:11:40.451350 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.450962    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d56d731-5280-4acc-bcda-a5eba4540f36" containerName="istio-proxy"
Nov 02 22:11:40.458638 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.458502    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"]
Nov 02 22:11:40.458638 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.458615    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:40.473163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.473138    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtcxd\" (UniqueName: \"kubernetes.io/projected/1b3014cf-9f59-4bd5-9d00-d8963e680bee-kube-api-access-wtcxd\") pod \"payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m\" (UID: \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:40.517609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.517575    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68345e06-0f8b-4885-ab52-02a3271750b3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8\" (UID: \"68345e06-0f8b-4885-ab52-02a3271750b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:40.517786 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.517753    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfbcr\" (UniqueName: \"kubernetes.io/projected/68345e06-0f8b-4885-ab52-02a3271750b3-kube-api-access-nfbcr\") pod \"payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8\" (UID: \"68345e06-0f8b-4885-ab52-02a3271750b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:40.544575 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.544324    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:40.622964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.621392    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68345e06-0f8b-4885-ab52-02a3271750b3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8\" (UID: \"68345e06-0f8b-4885-ab52-02a3271750b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:40.622964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.621605    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfbcr\" (UniqueName: \"kubernetes.io/projected/68345e06-0f8b-4885-ab52-02a3271750b3-kube-api-access-nfbcr\") pod \"payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8\" (UID: \"68345e06-0f8b-4885-ab52-02a3271750b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:40.622964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.622901    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68345e06-0f8b-4885-ab52-02a3271750b3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8\" (UID: \"68345e06-0f8b-4885-ab52-02a3271750b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:40.678482 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.678448    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfbcr\" (UniqueName: \"kubernetes.io/projected/68345e06-0f8b-4885-ab52-02a3271750b3-kube-api-access-nfbcr\") pod \"payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8\" (UID: \"68345e06-0f8b-4885-ab52-02a3271750b3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:40.776776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.776689    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:40.918315 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:11:40.918286    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3014cf_9f59_4bd5_9d00_d8963e680bee.slice/crio-34644f5caa05c286565e64c78e3c1d9fbdde6dbbb9b740ff81d1e5bb14d8e57f WatchSource:0}: Error finding container 34644f5caa05c286565e64c78e3c1d9fbdde6dbbb9b740ff81d1e5bb14d8e57f: Status 404 returned error can't find the container with id 34644f5caa05c286565e64c78e3c1d9fbdde6dbbb9b740ff81d1e5bb14d8e57f
Nov 02 22:11:40.921735 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:40.921714    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"]
Nov 02 22:11:41.148276 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:11:41.148241    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68345e06_0f8b_4885_ab52_02a3271750b3.slice/crio-99dc3fca7393b546cc3452695b91c3431a9a7caab37b7434060155437a7a3348 WatchSource:0}: Error finding container 99dc3fca7393b546cc3452695b91c3431a9a7caab37b7434060155437a7a3348: Status 404 returned error can't find the container with id 99dc3fca7393b546cc3452695b91c3431a9a7caab37b7434060155437a7a3348
Nov 02 22:11:41.150990 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.150967    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"]
Nov 02 22:11:41.287385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.287328    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52" podUID="d19974f7-0c80-4573-968d-350048867d30" containerName="payload-processing" containerID="cri-o://022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4" gracePeriod=25
Nov 02 22:11:41.393612 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.393564    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww" podUID="6cf37c52-d094-41b5-bfc7-48fab52419ee" containerName="payload-pre-processing" containerID="cri-o://58fb79312276d93c57c33acac94290a234c860ec6b6b2158f8ab21c46b9179e0" gracePeriod=25
Nov 02 22:11:41.568471 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.568445    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:11:41.572528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.572485    4528 generic.go:356] "Generic (PLEG): container finished" podID="d19974f7-0c80-4573-968d-350048867d30" containerID="022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4" exitCode=0
Nov 02 22:11:41.572667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.572567    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52" event={"ID":"d19974f7-0c80-4573-968d-350048867d30","Type":"ContainerDied","Data":"022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4"}
Nov 02 22:11:41.572667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.572632    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52" event={"ID":"d19974f7-0c80-4573-968d-350048867d30","Type":"ContainerDied","Data":"5d87836b44dd0f5ae1e3dc1a30dae7682d2e95ed326c1f266cab6e3fbc34d930"}
Nov 02 22:11:41.572667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.572660    4528 scope.go:122] "RemoveContainer" containerID="022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4"
Nov 02 22:11:41.572900 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.572883    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"
Nov 02 22:11:41.574734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.574710    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8" event={"ID":"68345e06-0f8b-4885-ab52-02a3271750b3","Type":"ContainerStarted","Data":"daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b"}
Nov 02 22:11:41.574852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.574759    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8" event={"ID":"68345e06-0f8b-4885-ab52-02a3271750b3","Type":"ContainerStarted","Data":"99dc3fca7393b546cc3452695b91c3431a9a7caab37b7434060155437a7a3348"}
Nov 02 22:11:41.575529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.575471    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:41.577700 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.577529    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m" event={"ID":"1b3014cf-9f59-4bd5-9d00-d8963e680bee","Type":"ContainerStarted","Data":"7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30"}
Nov 02 22:11:41.577700 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.577575    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m" event={"ID":"1b3014cf-9f59-4bd5-9d00-d8963e680bee","Type":"ContainerStarted","Data":"34644f5caa05c286565e64c78e3c1d9fbdde6dbbb9b740ff81d1e5bb14d8e57f"}
Nov 02 22:11:41.581324 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.578259    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:41.581324 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.580140    4528 generic.go:356] "Generic (PLEG): container finished" podID="6cf37c52-d094-41b5-bfc7-48fab52419ee" containerID="58fb79312276d93c57c33acac94290a234c860ec6b6b2158f8ab21c46b9179e0" exitCode=0
Nov 02 22:11:41.581324 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.580316    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww" event={"ID":"6cf37c52-d094-41b5-bfc7-48fab52419ee","Type":"ContainerDied","Data":"58fb79312276d93c57c33acac94290a234c860ec6b6b2158f8ab21c46b9179e0"}
Nov 02 22:11:41.595330 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.595299    4528 scope.go:122] "RemoveContainer" containerID="022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4"
Nov 02 22:11:41.596015 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:11:41.595986    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4\": container with ID starting with 022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4 not found: ID does not exist" containerID="022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4"
Nov 02 22:11:41.596144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.596022    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4"} err="failed to get container status \"022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4\": rpc error: code = NotFound desc = could not find container \"022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4\": container with ID starting with 022c7bd1b98656876e0faf22dd305a95720d91050bf6f7dfbf680312e1db8de4 not found: ID does not exist"
Nov 02 22:11:41.686427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.686403    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:11:41.732907 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.732871    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19974f7-0c80-4573-968d-350048867d30-kube-api-access-2dx6x\" (UniqueName: \"kubernetes.io/projected/d19974f7-0c80-4573-968d-350048867d30-kube-api-access-2dx6x\") pod \"d19974f7-0c80-4573-968d-350048867d30\" (UID: \"d19974f7-0c80-4573-968d-350048867d30\") "
Nov 02 22:11:41.733132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.732965    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf37c52-d094-41b5-bfc7-48fab52419ee-kube-api-access-b59ct\" (UniqueName: \"kubernetes.io/projected/6cf37c52-d094-41b5-bfc7-48fab52419ee-kube-api-access-b59ct\") pod \"6cf37c52-d094-41b5-bfc7-48fab52419ee\" (UID: \"6cf37c52-d094-41b5-bfc7-48fab52419ee\") "
Nov 02 22:11:41.733132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.733098    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf37c52-d094-41b5-bfc7-48fab52419ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf37c52-d094-41b5-bfc7-48fab52419ee-plugins-config-volume\") pod \"6cf37c52-d094-41b5-bfc7-48fab52419ee\" (UID: \"6cf37c52-d094-41b5-bfc7-48fab52419ee\") "
Nov 02 22:11:41.733285 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.733166    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19974f7-0c80-4573-968d-350048867d30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19974f7-0c80-4573-968d-350048867d30-plugins-config-volume\") pod \"d19974f7-0c80-4573-968d-350048867d30\" (UID: \"d19974f7-0c80-4573-968d-350048867d30\") "
Nov 02 22:11:41.733478 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.733455    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf37c52-d094-41b5-bfc7-48fab52419ee-plugins-config-volume" pod "6cf37c52-d094-41b5-bfc7-48fab52419ee" (UID: "6cf37c52-d094-41b5-bfc7-48fab52419ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:11:41.733581 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.733548    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19974f7-0c80-4573-968d-350048867d30-plugins-config-volume" pod "d19974f7-0c80-4573-968d-350048867d30" (UID: "d19974f7-0c80-4573-968d-350048867d30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:11:41.733696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.733675    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf37c52-d094-41b5-bfc7-48fab52419ee-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:41.733764 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.733702    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19974f7-0c80-4573-968d-350048867d30-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:41.735306 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.735274    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf37c52-d094-41b5-bfc7-48fab52419ee-kube-api-access-b59ct" pod "6cf37c52-d094-41b5-bfc7-48fab52419ee" (UID: "6cf37c52-d094-41b5-bfc7-48fab52419ee"). InnerVolumeSpecName "kube-api-access-b59ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:11:41.735403 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.735306    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19974f7-0c80-4573-968d-350048867d30-kube-api-access-2dx6x" pod "d19974f7-0c80-4573-968d-350048867d30" (UID: "d19974f7-0c80-4573-968d-350048867d30"). InnerVolumeSpecName "kube-api-access-2dx6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:11:41.795864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.795808    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m" podStartSLOduration=1.795793658 podStartE2EDuration="1.795793658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:41.79372324 +0000 UTC m=+3789.057259558" watchObservedRunningTime="2026-11-02 22:11:41.795793658 +0000 UTC m=+3789.059329973"
Nov 02 22:11:41.834989 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.834960    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dx6x\" (UniqueName: \"kubernetes.io/projected/d19974f7-0c80-4573-968d-350048867d30-kube-api-access-2dx6x\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:41.834989 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:41.834986    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b59ct\" (UniqueName: \"kubernetes.io/projected/6cf37c52-d094-41b5-bfc7-48fab52419ee-kube-api-access-b59ct\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:42.101618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.101557    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8" podStartSLOduration=2.101540743 podStartE2EDuration="2.101540743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:41.924345897 +0000 UTC m=+3789.187882218" watchObservedRunningTime="2026-11-02 22:11:42.101540743 +0000 UTC m=+3789.365077067"
Nov 02 22:11:42.211948 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.211916    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"]
Nov 02 22:11:42.214904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.214877    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5ddbd0-7765c985f9-xzx52"]
Nov 02 22:11:42.587544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.587514    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"
Nov 02 22:11:42.588001 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.587542    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww" event={"ID":"6cf37c52-d094-41b5-bfc7-48fab52419ee","Type":"ContainerDied","Data":"c691f5b04c816a8c88978c1708a101d14db5e4cd5728c2863a1a52c8491155c1"}
Nov 02 22:11:42.588001 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.587603    4528 scope.go:122] "RemoveContainer" containerID="58fb79312276d93c57c33acac94290a234c860ec6b6b2158f8ab21c46b9179e0"
Nov 02 22:11:42.752855 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.752754    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"]
Nov 02 22:11:42.756118 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.756088    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5ddbd0-6846487455-b66ww"]
Nov 02 22:11:42.914582 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.914552    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf37c52-d094-41b5-bfc7-48fab52419ee" path="/var/lib/kubelet/pods/6cf37c52-d094-41b5-bfc7-48fab52419ee/volumes"
Nov 02 22:11:42.914978 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:42.914964    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19974f7-0c80-4573-968d-350048867d30" path="/var/lib/kubelet/pods/d19974f7-0c80-4573-968d-350048867d30/volumes"
Nov 02 22:11:44.141255 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:11:44.141215    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:11:48.682671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:48.682635    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"]
Nov 02 22:11:53.600966 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:53.600932    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:11:53.601396 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:53.601005    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:11:53.682932 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:53.682883    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l" podUID="511810e8-c754-4efd-8224-991d37798c99" containerName="payload-processing" containerID="cri-o://b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18" gracePeriod=25
Nov 02 22:11:53.733702 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:11:53.733648    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511810e8_c754_4efd_8224_991d37798c99.slice/crio-b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 22:11:53.971699 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:53.971669    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:11:54.108241 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.108200    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511810e8-c754-4efd-8224-991d37798c99-kube-api-access-92879\" (UniqueName: \"kubernetes.io/projected/511810e8-c754-4efd-8224-991d37798c99-kube-api-access-92879\") pod \"511810e8-c754-4efd-8224-991d37798c99\" (UID: \"511810e8-c754-4efd-8224-991d37798c99\") "
Nov 02 22:11:54.108442 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.108323    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511810e8-c754-4efd-8224-991d37798c99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511810e8-c754-4efd-8224-991d37798c99-plugins-config-volume\") pod \"511810e8-c754-4efd-8224-991d37798c99\" (UID: \"511810e8-c754-4efd-8224-991d37798c99\") "
Nov 02 22:11:54.108694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.108669    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511810e8-c754-4efd-8224-991d37798c99-plugins-config-volume" pod "511810e8-c754-4efd-8224-991d37798c99" (UID: "511810e8-c754-4efd-8224-991d37798c99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:11:54.110506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.110479    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511810e8-c754-4efd-8224-991d37798c99-kube-api-access-92879" pod "511810e8-c754-4efd-8224-991d37798c99" (UID: "511810e8-c754-4efd-8224-991d37798c99"). InnerVolumeSpecName "kube-api-access-92879". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:11:54.209817 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.209782    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511810e8-c754-4efd-8224-991d37798c99-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:54.209817 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.209822    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92879\" (UniqueName: \"kubernetes.io/projected/511810e8-c754-4efd-8224-991d37798c99-kube-api-access-92879\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:11:54.684203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.684169    4528 generic.go:356] "Generic (PLEG): container finished" podID="511810e8-c754-4efd-8224-991d37798c99" containerID="b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18" exitCode=0
Nov 02 22:11:54.684617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.684238    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"
Nov 02 22:11:54.684617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.684254    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l" event={"ID":"511810e8-c754-4efd-8224-991d37798c99","Type":"ContainerDied","Data":"b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18"}
Nov 02 22:11:54.684617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.684337    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l" event={"ID":"511810e8-c754-4efd-8224-991d37798c99","Type":"ContainerDied","Data":"fe5bd71ae3ef00625906da05338fff6a5d73733cf1b6ae62f66bb8fe64991a52"}
Nov 02 22:11:54.684617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.684368    4528 scope.go:122] "RemoveContainer" containerID="b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18"
Nov 02 22:11:54.695227 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.695210    4528 scope.go:122] "RemoveContainer" containerID="b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18"
Nov 02 22:11:54.695491 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:11:54.695470    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18\": container with ID starting with b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18 not found: ID does not exist" containerID="b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18"
Nov 02 22:11:54.695558 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.695499    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18"} err="failed to get container status \"b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18\": rpc error: code = NotFound desc = could not find container \"b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18\": container with ID starting with b385bacb25ed414ce7baab4127c157aff787a865ad7024ff2d87f999c5801b18 not found: ID does not exist"
Nov 02 22:11:54.849334 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.849299    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"]
Nov 02 22:11:54.852669 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.852634    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-363e15-66b545cbf9-qsm6l"]
Nov 02 22:11:54.914166 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:54.914133    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511810e8-c754-4efd-8224-991d37798c99" path="/var/lib/kubelet/pods/511810e8-c754-4efd-8224-991d37798c99/volumes"
Nov 02 22:11:56.301964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:11:56.301926    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:12:14.114143 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:12:14.114080    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:12:43.047221 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:43.047187    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"]
Nov 02 22:12:43.097184 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:43.097152    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1838cdc8-vhr6t"]
Nov 02 22:12:44.114756 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:12:44.114718    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:12:44.914520 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:44.914483    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5158f93e-b3dc-438e-a87d-69798c0c5c50" path="/var/lib/kubelet/pods/5158f93e-b3dc-438e-a87d-69798c0c5c50/volumes"
Nov 02 22:12:47.589332 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:47.589294    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:12:54.046773 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:54.046740    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"]
Nov 02 22:12:54.096468 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:54.096435    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e28b1d-vpspv"]
Nov 02 22:12:54.914731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:54.914694    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="923a65af-abd7-4774-93f9-94597d6cfb9f" path="/var/lib/kubelet/pods/923a65af-abd7-4774-93f9-94597d6cfb9f/volumes"
Nov 02 22:12:58.791031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.790997    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"]
Nov 02 22:12:58.791657 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.791610    4528 cpu_manager.go:414] "containerMap: removing container" podUID="511810e8-c754-4efd-8224-991d37798c99" containerName="payload-processing"
Nov 02 22:12:58.791657 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.791626    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511810e8-c754-4efd-8224-991d37798c99" containerName="payload-processing"
Nov 02 22:12:58.791822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.791663    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d19974f7-0c80-4573-968d-350048867d30" containerName="payload-processing"
Nov 02 22:12:58.791822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.791671    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19974f7-0c80-4573-968d-350048867d30" containerName="payload-processing"
Nov 02 22:12:58.791822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.791690    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6cf37c52-d094-41b5-bfc7-48fab52419ee" containerName="payload-pre-processing"
Nov 02 22:12:58.791822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.791698    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf37c52-d094-41b5-bfc7-48fab52419ee" containerName="payload-pre-processing"
Nov 02 22:12:58.791983 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.791861    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19974f7-0c80-4573-968d-350048867d30" containerName="payload-processing"
Nov 02 22:12:58.791983 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.791875    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf37c52-d094-41b5-bfc7-48fab52419ee" containerName="payload-pre-processing"
Nov 02 22:12:58.791983 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.791886    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="511810e8-c754-4efd-8224-991d37798c99" containerName="payload-processing"
Nov 02 22:12:58.796634 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.796606    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.936044 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.935989    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-72f279-openshift-default-dockercfg-flv5n\""
Nov 02 22:12:58.947970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.947331    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-podinfo\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.947970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.947455    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313de020-6c5e-455c-b7a0-d1a15228e8a1-istiod-ca-cert\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.947970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.947562    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmng\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-kube-api-access-ctmng\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.947970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.947648    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-credential-socket\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.947970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.947704    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-data\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.947970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.947765    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-certs\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.947970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.947805    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-token\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.947970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.947834    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-socket\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.947970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.947876    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-envoy\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:58.954909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:58.954848    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"]
Nov 02 22:12:59.068066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.051427    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-credential-socket\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.068942 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.068851    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-data\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.069871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.069796    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-data\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.072270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.070167    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-certs\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.072270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.070226    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-token\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.072270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.070261    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-socket\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.072270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.070312    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-envoy\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.072270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.070373    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-podinfo\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.072270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.070560    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313de020-6c5e-455c-b7a0-d1a15228e8a1-istiod-ca-cert\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.072270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.070705    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmng\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-kube-api-access-ctmng\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.072270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.071582    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-certs\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.073239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.072987    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313de020-6c5e-455c-b7a0-d1a15228e8a1-istiod-ca-cert\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.073239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.073051    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-credential-socket\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.073239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.073084    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-socket\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.077196 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.076034    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-podinfo\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.077196 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.076501    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-envoy\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.133888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.133857    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmng\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-kube-api-access-ctmng\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.167434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.167403    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-token\") pod \"e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") " pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.411286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.411185    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:12:59.476002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.475970    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"]
Nov 02 22:12:59.492293 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.492002    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.531135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.531084    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"]
Nov 02 22:12:59.545007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.544973    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:12:59.545007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.544979    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:12:59.545288 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.545224    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:12:59.545347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.545297    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:12:59.577326 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.577289    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.577523 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.577436    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.577621 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.577595    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jds\" (UniqueName: \"kubernetes.io/projected/e37906a5-ad80-4cbf-bef4-2f6a36419de3-kube-api-access-q2jds\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.678515 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.678443    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.678673 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.678586    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.678742 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.678692    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jds\" (UniqueName: \"kubernetes.io/projected/e37906a5-ad80-4cbf-bef4-2f6a36419de3-kube-api-access-q2jds\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.679192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.679171    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.681018 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.680994    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.736781 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.736748    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jds\" (UniqueName: \"kubernetes.io/projected/e37906a5-ad80-4cbf-bef4-2f6a36419de3-kube-api-access-q2jds\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.809258 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.809223    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:12:59.834288 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:12:59.834254    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313de020_6c5e_455c_b7a0_d1a15228e8a1.slice/crio-2fe9943f631ac89b08a3a8e29ca5cc19321929436038cc5e9e09a4d348d4de55 WatchSource:0}: Error finding container 2fe9943f631ac89b08a3a8e29ca5cc19321929436038cc5e9e09a4d348d4de55: Status 404 returned error can't find the container with id 2fe9943f631ac89b08a3a8e29ca5cc19321929436038cc5e9e09a4d348d4de55
Nov 02 22:12:59.837510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.837475    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:12:59.837615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.837549    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:12:59.837615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.837495    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"]
Nov 02 22:12:59.837615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:12:59.837585    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:13:00.182178 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:00.182141    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" event={"ID":"313de020-6c5e-455c-b7a0-d1a15228e8a1","Type":"ContainerStarted","Data":"52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538"}
Nov 02 22:13:00.182404 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:00.182229    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" event={"ID":"313de020-6c5e-455c-b7a0-d1a15228e8a1","Type":"ContainerStarted","Data":"2fe9943f631ac89b08a3a8e29ca5cc19321929436038cc5e9e09a4d348d4de55"}
Nov 02 22:13:00.190854 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:13:00.190827    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37906a5_ad80_4cbf_bef4_2f6a36419de3.slice/crio-634dcf00e930a876a793f7ab85b67de48dfa5d6292e1753eb70c071c596556a2 WatchSource:0}: Error finding container 634dcf00e930a876a793f7ab85b67de48dfa5d6292e1753eb70c071c596556a2: Status 404 returned error can't find the container with id 634dcf00e930a876a793f7ab85b67de48dfa5d6292e1753eb70c071c596556a2
Nov 02 22:13:00.193267 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:00.193230    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"]
Nov 02 22:13:00.302299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:00.302246    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" podStartSLOduration=2.302229068 podStartE2EDuration="2.302229068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:00.298634217 +0000 UTC m=+3867.562170535" watchObservedRunningTime="2026-11-02 22:13:00.302229068 +0000 UTC m=+3867.565765385"
Nov 02 22:13:00.412525 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:00.412434    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:13:00.444404 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:00.443550    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:13:01.190654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:01.190626    4528 generic.go:356] "Generic (PLEG): container finished" podID="e37906a5-ad80-4cbf-bef4-2f6a36419de3" containerID="7f18d0384b8f9c73535474e1c4d25a8cb3e37d9c65aa5e9620665b3c7920f5cd" exitCode=0
Nov 02 22:13:01.191073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:01.190695    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz" event={"ID":"e37906a5-ad80-4cbf-bef4-2f6a36419de3","Type":"ContainerDied","Data":"7f18d0384b8f9c73535474e1c4d25a8cb3e37d9c65aa5e9620665b3c7920f5cd"}
Nov 02 22:13:01.191073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:01.190745    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz" event={"ID":"e37906a5-ad80-4cbf-bef4-2f6a36419de3","Type":"ContainerStarted","Data":"634dcf00e930a876a793f7ab85b67de48dfa5d6292e1753eb70c071c596556a2"}
Nov 02 22:13:01.191073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:01.190907    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:13:01.191913 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:01.191895    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:13:02.357047 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.357015    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:13:02.426152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.419666    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-xks-ca\") pod \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") "
Nov 02 22:13:02.426152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.419888    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37906a5-ad80-4cbf-bef4-2f6a36419de3-kube-api-access-q2jds\" (UniqueName: \"kubernetes.io/projected/e37906a5-ad80-4cbf-bef4-2f6a36419de3-kube-api-access-q2jds\") pod \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") "
Nov 02 22:13:02.426152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.420002    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-openshift-ca\") pod \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\" (UID: \"e37906a5-ad80-4cbf-bef4-2f6a36419de3\") "
Nov 02 22:13:02.426152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.421102    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-openshift-ca" pod "e37906a5-ad80-4cbf-bef4-2f6a36419de3" (UID: "e37906a5-ad80-4cbf-bef4-2f6a36419de3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:13:02.442133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.433025    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37906a5-ad80-4cbf-bef4-2f6a36419de3-kube-api-access-q2jds" pod "e37906a5-ad80-4cbf-bef4-2f6a36419de3" (UID: "e37906a5-ad80-4cbf-bef4-2f6a36419de3"). InnerVolumeSpecName "kube-api-access-q2jds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:13:02.451580 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.451484    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-xks-ca" pod "e37906a5-ad80-4cbf-bef4-2f6a36419de3" (UID: "e37906a5-ad80-4cbf-bef4-2f6a36419de3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:13:02.521672 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.521639    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2jds\" (UniqueName: \"kubernetes.io/projected/e37906a5-ad80-4cbf-bef4-2f6a36419de3-kube-api-access-q2jds\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:02.521672 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.521666    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:02.521672 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:02.521675    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37906a5-ad80-4cbf-bef4-2f6a36419de3-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:03.207712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:03.207630    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"
Nov 02 22:13:03.207712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:03.207664    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz" event={"ID":"e37906a5-ad80-4cbf-bef4-2f6a36419de3","Type":"ContainerDied","Data":"634dcf00e930a876a793f7ab85b67de48dfa5d6292e1753eb70c071c596556a2"}
Nov 02 22:13:03.207906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:03.207751    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="634dcf00e930a876a793f7ab85b67de48dfa5d6292e1753eb70c071c596556a2"
Nov 02 22:13:06.554376 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.554337    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"]
Nov 02 22:13:06.555067 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.555050    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e37906a5-ad80-4cbf-bef4-2f6a36419de3" containerName="revoke-keys"
Nov 02 22:13:06.555142 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.555067    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37906a5-ad80-4cbf-bef4-2f6a36419de3" containerName="revoke-keys"
Nov 02 22:13:06.555190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.555182    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37906a5-ad80-4cbf-bef4-2f6a36419de3" containerName="revoke-keys"
Nov 02 22:13:06.558939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.558912    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:06.613857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.613806    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"]
Nov 02 22:13:06.685967 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.665433    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-72f279-dockercfg-lxlpj\""
Nov 02 22:13:06.685967 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.665524    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-72f279\""
Nov 02 22:13:06.734520 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.734470    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afb3dbf-9f78-482c-b663-9efefcf18168-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4\" (UID: \"3afb3dbf-9f78-482c-b663-9efefcf18168\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:06.734977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.734951    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g84zg\" (UniqueName: \"kubernetes.io/projected/3afb3dbf-9f78-482c-b663-9efefcf18168-kube-api-access-g84zg\") pod \"payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4\" (UID: \"3afb3dbf-9f78-482c-b663-9efefcf18168\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:06.753748 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.753708    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"]
Nov 02 22:13:06.760059 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.760027    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:06.786816 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.786785    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"]
Nov 02 22:13:06.837075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.836991    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db93c8f-772f-42f0-8d7e-beabf47d06d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72f279-645d65f547-wcwph\" (UID: \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:06.837271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.837101    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84zg\" (UniqueName: \"kubernetes.io/projected/3afb3dbf-9f78-482c-b663-9efefcf18168-kube-api-access-g84zg\") pod \"payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4\" (UID: \"3afb3dbf-9f78-482c-b663-9efefcf18168\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:06.837271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.837175    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9hjs\" (UniqueName: \"kubernetes.io/projected/9db93c8f-772f-42f0-8d7e-beabf47d06d9-kube-api-access-d9hjs\") pod \"payload-processing-e2e-inf-a-72f279-645d65f547-wcwph\" (UID: \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:06.837398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.837281    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afb3dbf-9f78-482c-b663-9efefcf18168-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4\" (UID: \"3afb3dbf-9f78-482c-b663-9efefcf18168\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:06.837946 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.837922    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afb3dbf-9f78-482c-b663-9efefcf18168-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4\" (UID: \"3afb3dbf-9f78-482c-b663-9efefcf18168\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:06.894039 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.894002    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84zg\" (UniqueName: \"kubernetes.io/projected/3afb3dbf-9f78-482c-b663-9efefcf18168-kube-api-access-g84zg\") pod \"payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4\" (UID: \"3afb3dbf-9f78-482c-b663-9efefcf18168\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:06.938305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.938267    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db93c8f-772f-42f0-8d7e-beabf47d06d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72f279-645d65f547-wcwph\" (UID: \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:06.938533 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.938359    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9hjs\" (UniqueName: \"kubernetes.io/projected/9db93c8f-772f-42f0-8d7e-beabf47d06d9-kube-api-access-d9hjs\") pod \"payload-processing-e2e-inf-a-72f279-645d65f547-wcwph\" (UID: \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:06.939031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.939010    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db93c8f-772f-42f0-8d7e-beabf47d06d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72f279-645d65f547-wcwph\" (UID: \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:06.995044 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:06.995014    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9hjs\" (UniqueName: \"kubernetes.io/projected/9db93c8f-772f-42f0-8d7e-beabf47d06d9-kube-api-access-d9hjs\") pod \"payload-processing-e2e-inf-a-72f279-645d65f547-wcwph\" (UID: \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:07.087432 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:07.087345    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:07.174742 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:07.173627    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:07.468457 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:13:07.468429    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db93c8f_772f_42f0_8d7e_beabf47d06d9.slice/crio-8a65caf66b122a9477f7e4c426c925926cf72f5e9951c5dca86ea0487a305009 WatchSource:0}: Error finding container 8a65caf66b122a9477f7e4c426c925926cf72f5e9951c5dca86ea0487a305009: Status 404 returned error can't find the container with id 8a65caf66b122a9477f7e4c426c925926cf72f5e9951c5dca86ea0487a305009
Nov 02 22:13:07.471947 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:07.471854    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"]
Nov 02 22:13:07.588317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:07.585344    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"]
Nov 02 22:13:08.244344 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:08.244313    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph" event={"ID":"9db93c8f-772f-42f0-8d7e-beabf47d06d9","Type":"ContainerStarted","Data":"abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd"}
Nov 02 22:13:08.244567 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:08.244361    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph" event={"ID":"9db93c8f-772f-42f0-8d7e-beabf47d06d9","Type":"ContainerStarted","Data":"8a65caf66b122a9477f7e4c426c925926cf72f5e9951c5dca86ea0487a305009"}
Nov 02 22:13:08.244567 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:08.244523    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:08.246168 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:08.246144    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4" event={"ID":"3afb3dbf-9f78-482c-b663-9efefcf18168","Type":"ContainerStarted","Data":"b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e"}
Nov 02 22:13:08.246261 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:08.246188    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4" event={"ID":"3afb3dbf-9f78-482c-b663-9efefcf18168","Type":"ContainerStarted","Data":"3b9d5491e272ebdca568ffba39adb7a0660d2ce8ea5cb90815a37aeaf9c3597f"}
Nov 02 22:13:08.246320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:08.246262    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:08.395694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:08.392387    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph" podStartSLOduration=2.392364484 podStartE2EDuration="2.392364484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:08.370814633 +0000 UTC m=+3875.634350952" watchObservedRunningTime="2026-11-02 22:13:08.392364484 +0000 UTC m=+3875.655900802"
Nov 02 22:13:08.495041 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:08.486310    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4" podStartSLOduration=2.4862946 podStartE2EDuration="2.4862946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:08.484150163 +0000 UTC m=+3875.747686485" watchObservedRunningTime="2026-11-02 22:13:08.4862946 +0000 UTC m=+3875.749830917"
Nov 02 22:13:11.749303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:11.749263    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"]
Nov 02 22:13:11.749721 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:11.749491    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy" containerID="cri-o://b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e" gracePeriod=30
Nov 02 22:13:14.236643 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:13:14.236605    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:13:14.290377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.290348    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"]
Nov 02 22:13:14.294950 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.294922    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.347951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.347916    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d6b384-openshift-default-dockercfg-8g8h8\""
Nov 02 22:13:14.403843 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.403808    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"]
Nov 02 22:13:14.419709 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.419670    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-token\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.419909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.419724    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-credential-socket\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.419909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.419821    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8cf2\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-kube-api-access-p8cf2\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.419909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.419856    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-certs\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.419909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.419888    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-socket\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.420293 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.420043    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-data\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.420293 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.420125    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-envoy\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.420293 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.420155    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-podinfo\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.420293 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.420253    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istiod-ca-cert\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.521980 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.521949    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8cf2\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-kube-api-access-p8cf2\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.521998    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-certs\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522030    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-socket\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522130    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-data\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522164    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-envoy\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522190    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-podinfo\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522283    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istiod-ca-cert\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522367    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-token\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522394    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-credential-socket\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522474    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-certs\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522492    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-socket\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522575    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-data\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.522782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.522734    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-credential-socket\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.523214 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.523191    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istiod-ca-cert\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.524591 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.524567    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-envoy\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.524879 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.524863    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-podinfo\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.579421 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.579389    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-token\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.579569 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.579528    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8cf2\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-kube-api-access-p8cf2\") pod \"e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") " pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:14.610568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:14.610539    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:15.065130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:15.062882    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"]
Nov 02 22:13:15.079200 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:15.079157    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:13:15.079361 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:15.079250    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:13:15.079361 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:15.079296    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:13:15.300181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:15.300139    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" event={"ID":"50aa6c40-0fa9-430a-94ea-12b06cb46bd8","Type":"ContainerStarted","Data":"696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab"}
Nov 02 22:13:15.300653 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:15.300202    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" event={"ID":"50aa6c40-0fa9-430a-94ea-12b06cb46bd8","Type":"ContainerStarted","Data":"6409b9d28376dedbc0b39c9fc64ba5e7f2c2e5688f79be1d10ec5f28945e8501"}
Nov 02 22:13:15.420330 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:15.420217    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" podStartSLOduration=1.42019733 podStartE2EDuration="1.42019733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:15.415476547 +0000 UTC m=+3882.679012887" watchObservedRunningTime="2026-11-02 22:13:15.42019733 +0000 UTC m=+3882.683733649"
Nov 02 22:13:15.611263 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:15.611233    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:16.616682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:16.616650    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:17.027808 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.027778    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:13:17.204278 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204194    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd786b7-e883-43a6-b539-c7c152512c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd786b7-e883-43a6-b539-c7c152512c0a-istiod-ca-cert\") pod \"bfd786b7-e883-43a6-b539-c7c152512c0a\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") "
Nov 02 22:13:17.204278 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204255    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-socket\") pod \"bfd786b7-e883-43a6-b539-c7c152512c0a\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") "
Nov 02 22:13:17.204579 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204303    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-credential-socket\") pod \"bfd786b7-e883-43a6-b539-c7c152512c0a\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") "
Nov 02 22:13:17.204579 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204357    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-token\") pod \"bfd786b7-e883-43a6-b539-c7c152512c0a\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") "
Nov 02 22:13:17.204579 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204483    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-kube-api-access-vdtz5\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-kube-api-access-vdtz5\") pod \"bfd786b7-e883-43a6-b539-c7c152512c0a\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") "
Nov 02 22:13:17.204750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204578    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-podinfo\") pod \"bfd786b7-e883-43a6-b539-c7c152512c0a\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") "
Nov 02 22:13:17.204750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204582    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-socket" pod "bfd786b7-e883-43a6-b539-c7c152512c0a" (UID: "bfd786b7-e883-43a6-b539-c7c152512c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:13:17.204750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204648    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd786b7-e883-43a6-b539-c7c152512c0a-istiod-ca-cert" pod "bfd786b7-e883-43a6-b539-c7c152512c0a" (UID: "bfd786b7-e883-43a6-b539-c7c152512c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:13:17.204750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204656    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-envoy\") pod \"bfd786b7-e883-43a6-b539-c7c152512c0a\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") "
Nov 02 22:13:17.204750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204725    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-data\") pod \"bfd786b7-e883-43a6-b539-c7c152512c0a\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") "
Nov 02 22:13:17.205045 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.204795    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-certs\") pod \"bfd786b7-e883-43a6-b539-c7c152512c0a\" (UID: \"bfd786b7-e883-43a6-b539-c7c152512c0a\") "
Nov 02 22:13:17.205105 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.205066    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-data" pod "bfd786b7-e883-43a6-b539-c7c152512c0a" (UID: "bfd786b7-e883-43a6-b539-c7c152512c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:13:17.205337 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.205311    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-certs" pod "bfd786b7-e883-43a6-b539-c7c152512c0a" (UID: "bfd786b7-e883-43a6-b539-c7c152512c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:13:17.205701 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.205677    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:17.205819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.205704    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:17.205819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.205721    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd786b7-e883-43a6-b539-c7c152512c0a-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:17.205819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.205735    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:17.206002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.205902    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-credential-socket" pod "bfd786b7-e883-43a6-b539-c7c152512c0a" (UID: "bfd786b7-e883-43a6-b539-c7c152512c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:13:17.206958 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.206931    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-podinfo" pod "bfd786b7-e883-43a6-b539-c7c152512c0a" (UID: "bfd786b7-e883-43a6-b539-c7c152512c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:13:17.207073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.207031    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-envoy" pod "bfd786b7-e883-43a6-b539-c7c152512c0a" (UID: "bfd786b7-e883-43a6-b539-c7c152512c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:13:17.207207 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.207192    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-token" pod "bfd786b7-e883-43a6-b539-c7c152512c0a" (UID: "bfd786b7-e883-43a6-b539-c7c152512c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:13:17.207630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.207616    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-kube-api-access-vdtz5" pod "bfd786b7-e883-43a6-b539-c7c152512c0a" (UID: "bfd786b7-e883-43a6-b539-c7c152512c0a"). InnerVolumeSpecName "kube-api-access-vdtz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:13:17.307702 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.307663    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:17.307702 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.307700    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:17.307956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.307715    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdtz5\" (UniqueName: \"kubernetes.io/projected/bfd786b7-e883-43a6-b539-c7c152512c0a-kube-api-access-vdtz5\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:17.307956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.307730    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:17.307956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.307743    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd786b7-e883-43a6-b539-c7c152512c0a-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:13:17.315618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.315590    4528 generic.go:356] "Generic (PLEG): container finished" podID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerID="b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e" exitCode=0
Nov 02 22:13:17.315738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.315640    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"
Nov 02 22:13:17.315738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.315674    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" event={"ID":"bfd786b7-e883-43a6-b539-c7c152512c0a","Type":"ContainerDied","Data":"b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e"}
Nov 02 22:13:17.315738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.315730    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr" event={"ID":"bfd786b7-e883-43a6-b539-c7c152512c0a","Type":"ContainerDied","Data":"f474aa6cec389855250090319d74c35532f68ec9e45cf743b1f37f55e9f7f1cf"}
Nov 02 22:13:17.315882 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.315759    4528 scope.go:122] "RemoveContainer" containerID="b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e"
Nov 02 22:13:17.316162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.316064    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:17.317034 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.317016    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:13:17.330232 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.330208    4528 scope.go:122] "RemoveContainer" containerID="cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96"
Nov 02 22:13:17.339587 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.339565    4528 scope.go:122] "RemoveContainer" containerID="b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e"
Nov 02 22:13:17.339836 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:13:17.339819    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e\": container with ID starting with b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e not found: ID does not exist" containerID="b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e"
Nov 02 22:13:17.339904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.339843    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e"} err="failed to get container status \"b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e\": rpc error: code = NotFound desc = could not find container \"b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e\": container with ID starting with b204c287ad33fb6a01a580ea664fab417ca86c941731d2aea202faec550fb47e not found: ID does not exist"
Nov 02 22:13:17.339904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.339864    4528 scope.go:122] "RemoveContainer" containerID="cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96"
Nov 02 22:13:17.340603 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:13:17.340568    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96\": container with ID starting with cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96 not found: ID does not exist" containerID="cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96"
Nov 02 22:13:17.340710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.340624    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96"} err="failed to get container status \"cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96\": rpc error: code = NotFound desc = could not find container \"cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96\": container with ID starting with cdec06f34b5351e76efcbee72601c921f87f9fe212cdcab4e2bb58235ec7ee96 not found: ID does not exist"
Nov 02 22:13:17.482859 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.482776    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"]
Nov 02 22:13:17.487945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:17.487914    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-363e15-openshift-default-754479d777-k44vr"]
Nov 02 22:13:18.914969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:18.914937    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" path="/var/lib/kubelet/pods/bfd786b7-e883-43a6-b539-c7c152512c0a/volumes"
Nov 02 22:13:19.263307 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:19.263280    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:13:19.263495 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:19.263349    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:13:21.690035 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.689999    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"]
Nov 02 22:13:21.690686 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.690671    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy"
Nov 02 22:13:21.690738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.690687    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy"
Nov 02 22:13:21.690738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.690713    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy"
Nov 02 22:13:21.690738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.690718    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy"
Nov 02 22:13:21.690861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.690824    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy"
Nov 02 22:13:21.690861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.690836    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd786b7-e883-43a6-b539-c7c152512c0a" containerName="istio-proxy"
Nov 02 22:13:21.696283 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.696259    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:21.747514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.747474    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"]
Nov 02 22:13:21.803129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.800482    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d6b384-dockercfg-hgv4q\""
Nov 02 22:13:21.803129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.800589    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d6b384\""
Nov 02 22:13:21.879133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.876966    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a78a940-be05-41d1-808d-d7df4d3b076b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq\" (UID: \"8a78a940-be05-41d1-808d-d7df4d3b076b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:21.879133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.877148    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvdgx\" (UniqueName: \"kubernetes.io/projected/8a78a940-be05-41d1-808d-d7df4d3b076b-kube-api-access-fvdgx\") pod \"payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq\" (UID: \"8a78a940-be05-41d1-808d-d7df4d3b076b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:21.948339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.948251    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"]
Nov 02 22:13:21.954551 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.954525    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"]
Nov 02 22:13:21.954731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.954659    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:21.978816 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.978782    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvdgx\" (UniqueName: \"kubernetes.io/projected/8a78a940-be05-41d1-808d-d7df4d3b076b-kube-api-access-fvdgx\") pod \"payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq\" (UID: \"8a78a940-be05-41d1-808d-d7df4d3b076b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:21.979014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.978849    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8rm\" (UniqueName: \"kubernetes.io/projected/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-kube-api-access-6q8rm\") pod \"payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7\" (UID: \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:21.979014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.978915    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7\" (UID: \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:21.979180 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.979151    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a78a940-be05-41d1-808d-d7df4d3b076b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq\" (UID: \"8a78a940-be05-41d1-808d-d7df4d3b076b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:21.979949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:21.979929    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a78a940-be05-41d1-808d-d7df4d3b076b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq\" (UID: \"8a78a940-be05-41d1-808d-d7df4d3b076b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:22.035716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:22.035684    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvdgx\" (UniqueName: \"kubernetes.io/projected/8a78a940-be05-41d1-808d-d7df4d3b076b-kube-api-access-fvdgx\") pod \"payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq\" (UID: \"8a78a940-be05-41d1-808d-d7df4d3b076b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:22.080080 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:22.080038    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8rm\" (UniqueName: \"kubernetes.io/projected/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-kube-api-access-6q8rm\") pod \"payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7\" (UID: \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:22.080344 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:22.080154    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7\" (UID: \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:22.081080 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:22.081054    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7\" (UID: \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:22.137055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:22.137017    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8rm\" (UniqueName: \"kubernetes.io/projected/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-kube-api-access-6q8rm\") pod \"payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7\" (UID: \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:22.270671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:22.270633    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:22.310014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:22.309979    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:22.645300 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:13:22.645265    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1ba9d7_47b4_4779_9ecc_a35d9ae5a764.slice/crio-d0207f9a774cf1900e666a0e25640b45c0834014267f4f87b3e185253de5a23d WatchSource:0}: Error finding container d0207f9a774cf1900e666a0e25640b45c0834014267f4f87b3e185253de5a23d: Status 404 returned error can't find the container with id d0207f9a774cf1900e666a0e25640b45c0834014267f4f87b3e185253de5a23d
Nov 02 22:13:22.648568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:22.648544    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"]
Nov 02 22:13:22.741297 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:13:22.741260    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a78a940_be05_41d1_808d_d7df4d3b076b.slice/crio-fd38ec22084a106a70fb3874efa0a4fd3afa9abd8e32be8ccfa198c1e13d71ca WatchSource:0}: Error finding container fd38ec22084a106a70fb3874efa0a4fd3afa9abd8e32be8ccfa198c1e13d71ca: Status 404 returned error can't find the container with id fd38ec22084a106a70fb3874efa0a4fd3afa9abd8e32be8ccfa198c1e13d71ca
Nov 02 22:13:22.744275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:22.744253    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"]
Nov 02 22:13:23.380300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:23.380259    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq" event={"ID":"8a78a940-be05-41d1-808d-d7df4d3b076b","Type":"ContainerStarted","Data":"80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c"}
Nov 02 22:13:23.380541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:23.380321    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq" event={"ID":"8a78a940-be05-41d1-808d-d7df4d3b076b","Type":"ContainerStarted","Data":"fd38ec22084a106a70fb3874efa0a4fd3afa9abd8e32be8ccfa198c1e13d71ca"}
Nov 02 22:13:23.380541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:23.380472    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:23.382241 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:23.382212    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7" event={"ID":"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764","Type":"ContainerStarted","Data":"127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37"}
Nov 02 22:13:23.382372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:23.382257    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7" event={"ID":"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764","Type":"ContainerStarted","Data":"d0207f9a774cf1900e666a0e25640b45c0834014267f4f87b3e185253de5a23d"}
Nov 02 22:13:23.382372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:23.382295    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:23.493945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:23.493881    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq" podStartSLOduration=2.493865544 podStartE2EDuration="2.493865544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:23.492339082 +0000 UTC m=+3890.755875402" watchObservedRunningTime="2026-11-02 22:13:23.493865544 +0000 UTC m=+3890.757401876"
Nov 02 22:13:23.626179 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:23.621477    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7" podStartSLOduration=2.621454885 podStartE2EDuration="2.621454885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:23.620831135 +0000 UTC m=+3890.884367476" watchObservedRunningTime="2026-11-02 22:13:23.621454885 +0000 UTC m=+3890.884991206"
Nov 02 22:13:34.391997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:34.391955    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:13:34.392558 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:34.392359    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:13:40.407671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:40.407633    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:13:44.115082 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:13:44.115045    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:13:57.921928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:57.921889    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"]
Nov 02 22:13:58.031296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:13:58.031258    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"]
Nov 02 22:14:01.684956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:01.684905    4528 generic.go:356] "Generic (PLEG): container finished" podID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerID="b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b" exitCode=0
Nov 02 22:14:01.685437 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:01.684974    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" event={"ID":"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70","Type":"ContainerDied","Data":"b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b"}
Nov 02 22:14:01.685528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:01.685511    4528 scope.go:122] "RemoveContainer" containerID="b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b"
Nov 02 22:14:01.687340 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:01.687308    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:14:01.687474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:01.687387    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:14:01.687474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:01.687422    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:14:01.918210 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:01.918176    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:14:02.693955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:02.693912    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" event={"ID":"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70","Type":"ContainerStarted","Data":"95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e"}
Nov 02 22:14:02.878781 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:02.878722    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:14:02.879954 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:02.879931    4528 patch_prober.go:29] interesting pod/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 02 22:14:02.880018 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:02.879991    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 02 22:14:02.927250 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:02.927202    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj" podUID="f8f72523-e582-4041-aa7b-159b17143c40" containerName="payload-processing" containerID="cri-o://b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253" gracePeriod=25
Nov 02 22:14:03.032816 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.032768    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx" podUID="a9ae8f91-cece-4328-a38f-fc8cdb44607a" containerName="payload-pre-processing" containerID="cri-o://d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4" gracePeriod=25
Nov 02 22:14:03.281651 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.281237    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:14:03.363977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.363953    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:14:03.425135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.424561    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f72523-e582-4041-aa7b-159b17143c40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f72523-e582-4041-aa7b-159b17143c40-plugins-config-volume\") pod \"f8f72523-e582-4041-aa7b-159b17143c40\" (UID: \"f8f72523-e582-4041-aa7b-159b17143c40\") "
Nov 02 22:14:03.425135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.424639    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f72523-e582-4041-aa7b-159b17143c40-kube-api-access-ffdm8\" (UniqueName: \"kubernetes.io/projected/f8f72523-e582-4041-aa7b-159b17143c40-kube-api-access-ffdm8\") pod \"f8f72523-e582-4041-aa7b-159b17143c40\" (UID: \"f8f72523-e582-4041-aa7b-159b17143c40\") "
Nov 02 22:14:03.427567 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.427519    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f72523-e582-4041-aa7b-159b17143c40-plugins-config-volume" pod "f8f72523-e582-4041-aa7b-159b17143c40" (UID: "f8f72523-e582-4041-aa7b-159b17143c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:03.433265 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.431010    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f72523-e582-4041-aa7b-159b17143c40-kube-api-access-ffdm8" pod "f8f72523-e582-4041-aa7b-159b17143c40" (UID: "f8f72523-e582-4041-aa7b-159b17143c40"). InnerVolumeSpecName "kube-api-access-ffdm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:03.533465 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.533428    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ae8f91-cece-4328-a38f-fc8cdb44607a-kube-api-access-jns2g\" (UniqueName: \"kubernetes.io/projected/a9ae8f91-cece-4328-a38f-fc8cdb44607a-kube-api-access-jns2g\") pod \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\" (UID: \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\") "
Nov 02 22:14:03.533680 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.533515    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ae8f91-cece-4328-a38f-fc8cdb44607a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ae8f91-cece-4328-a38f-fc8cdb44607a-plugins-config-volume\") pod \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\" (UID: \"a9ae8f91-cece-4328-a38f-fc8cdb44607a\") "
Nov 02 22:14:03.534154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.534129    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f72523-e582-4041-aa7b-159b17143c40-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:03.534277 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.534157    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffdm8\" (UniqueName: \"kubernetes.io/projected/f8f72523-e582-4041-aa7b-159b17143c40-kube-api-access-ffdm8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:03.534527 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.534503    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ae8f91-cece-4328-a38f-fc8cdb44607a-plugins-config-volume" pod "a9ae8f91-cece-4328-a38f-fc8cdb44607a" (UID: "a9ae8f91-cece-4328-a38f-fc8cdb44607a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:03.537002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.536958    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ae8f91-cece-4328-a38f-fc8cdb44607a-kube-api-access-jns2g" pod "a9ae8f91-cece-4328-a38f-fc8cdb44607a" (UID: "a9ae8f91-cece-4328-a38f-fc8cdb44607a"). InnerVolumeSpecName "kube-api-access-jns2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:03.635526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.635482    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jns2g\" (UniqueName: \"kubernetes.io/projected/a9ae8f91-cece-4328-a38f-fc8cdb44607a-kube-api-access-jns2g\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:03.635526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.635521    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ae8f91-cece-4328-a38f-fc8cdb44607a-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:03.726189 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.724598    4528 generic.go:356] "Generic (PLEG): container finished" podID="f8f72523-e582-4041-aa7b-159b17143c40" containerID="b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253" exitCode=0
Nov 02 22:14:03.726189 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.724769    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj" event={"ID":"f8f72523-e582-4041-aa7b-159b17143c40","Type":"ContainerDied","Data":"b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253"}
Nov 02 22:14:03.726189 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.724823    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj" event={"ID":"f8f72523-e582-4041-aa7b-159b17143c40","Type":"ContainerDied","Data":"9661ceb492c01c1dcf668a9b434838aee4fb0143e0631d65e19b4a06321a11f9"}
Nov 02 22:14:03.726189 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.724854    4528 scope.go:122] "RemoveContainer" containerID="b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253"
Nov 02 22:14:03.726189 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.725045    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"
Nov 02 22:14:03.729014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.728979    4528 generic.go:356] "Generic (PLEG): container finished" podID="a9ae8f91-cece-4328-a38f-fc8cdb44607a" containerID="d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4" exitCode=0
Nov 02 22:14:03.729164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.729025    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"
Nov 02 22:14:03.729253 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.729193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx" event={"ID":"a9ae8f91-cece-4328-a38f-fc8cdb44607a","Type":"ContainerDied","Data":"d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4"}
Nov 02 22:14:03.729323 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.729256    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx" event={"ID":"a9ae8f91-cece-4328-a38f-fc8cdb44607a","Type":"ContainerDied","Data":"d84bb1d7905b3f3564d93647c17a50861052e4ff606124791c3ab20893eb6ed1"}
Nov 02 22:14:03.738973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.738950    4528 scope.go:122] "RemoveContainer" containerID="b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253"
Nov 02 22:14:03.739327 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:14:03.739308    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253\": container with ID starting with b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253 not found: ID does not exist" containerID="b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253"
Nov 02 22:14:03.739427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.739334    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253"} err="failed to get container status \"b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253\": rpc error: code = NotFound desc = could not find container \"b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253\": container with ID starting with b31cb5ba92c56f5ec126ff4af8b66bf37819966cab305a0200412345b3eac253 not found: ID does not exist"
Nov 02 22:14:03.739427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.739351    4528 scope.go:122] "RemoveContainer" containerID="d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4"
Nov 02 22:14:03.749573 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.749551    4528 scope.go:122] "RemoveContainer" containerID="d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4"
Nov 02 22:14:03.749851 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:14:03.749831    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4\": container with ID starting with d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4 not found: ID does not exist" containerID="d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4"
Nov 02 22:14:03.749967 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.749859    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4"} err="failed to get container status \"d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4\": rpc error: code = NotFound desc = could not find container \"d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4\": container with ID starting with d1c0d377dbb882de94892753fc491cefdb12a31fdd483a564415e7a12144e3d4 not found: ID does not exist"
Nov 02 22:14:03.879052 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.878971    4528 patch_prober.go:29] interesting pod/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 02 22:14:03.879052 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.879036    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 02 22:14:03.892325 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.892292    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"]
Nov 02 22:14:03.895311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:03.895290    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-065be7-56956665f5-psbzx"]
Nov 02 22:14:04.027134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:04.026188    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"]
Nov 02 22:14:04.027837 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:04.027815    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-065be7-5b8f94455-6h8jj"]
Nov 02 22:14:04.879324 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:04.879281    4528 patch_prober.go:29] interesting pod/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 02 22:14:04.879835 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:04.879353    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 02 22:14:04.932685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:04.932651    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ae8f91-cece-4328-a38f-fc8cdb44607a" path="/var/lib/kubelet/pods/a9ae8f91-cece-4328-a38f-fc8cdb44607a/volumes"
Nov 02 22:14:04.933170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:04.933155    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f72523-e582-4041-aa7b-159b17143c40" path="/var/lib/kubelet/pods/f8f72523-e582-4041-aa7b-159b17143c40/volumes"
Nov 02 22:14:05.879657 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:05.879619    4528 patch_prober.go:29] interesting pod/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 02 22:14:05.880061 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:05.879680    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 02 22:14:06.879376 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:06.879337    4528 patch_prober.go:29] interesting pod/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 02 22:14:06.879604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:06.879399    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 02 22:14:07.879310 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:07.879277    4528 patch_prober.go:29] interesting pod/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 02 22:14:07.879761 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:07.879332    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 02 22:14:08.884029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:08.883997    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:14:08.884484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:08.884324    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:14:08.885151 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:08.885130    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:14:14.117651 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:14:14.117613    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:14:36.097193 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:36.097158    4528 scope.go:122] "RemoveContainer" containerID="dfda084fcac97b09af08d47cfc92ef888c1f6d299f739bb803ec40de32203e32"
Nov 02 22:14:36.107574 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:36.107551    4528 scope.go:122] "RemoveContainer" containerID="b32036db68257273b00ab563eaa829fe67a9632129b95638151a53be3d345bd7"
Nov 02 22:14:39.161945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:39.161905    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"]
Nov 02 22:14:39.211872 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:39.211834    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5ddbd0-tk5vz"]
Nov 02 22:14:40.915519 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:40.915488    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37906a5-ad80-4cbf-bef4-2f6a36419de3" path="/var/lib/kubelet/pods/e37906a5-ad80-4cbf-bef4-2f6a36419de3/volumes"
Nov 02 22:14:44.113429 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:14:44.113351    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:14:44.136758 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:44.136723    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"]
Nov 02 22:14:44.242852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:44.242816    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"]
Nov 02 22:14:47.761745 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:47.761704    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:14:49.157223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.157174    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8" podUID="68345e06-0f8b-4885-ab52-02a3271750b3" containerName="payload-processing" containerID="cri-o://daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b" gracePeriod=25
Nov 02 22:14:49.215427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.215388    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"]
Nov 02 22:14:49.216485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.216459    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f8f72523-e582-4041-aa7b-159b17143c40" containerName="payload-processing"
Nov 02 22:14:49.216620 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.216487    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f72523-e582-4041-aa7b-159b17143c40" containerName="payload-processing"
Nov 02 22:14:49.216620 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.216536    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a9ae8f91-cece-4328-a38f-fc8cdb44607a" containerName="payload-pre-processing"
Nov 02 22:14:49.216620 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.216546    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ae8f91-cece-4328-a38f-fc8cdb44607a" containerName="payload-pre-processing"
Nov 02 22:14:49.216808 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.216694    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ae8f91-cece-4328-a38f-fc8cdb44607a" containerName="payload-pre-processing"
Nov 02 22:14:49.216808 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.216717    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f72523-e582-4041-aa7b-159b17143c40" containerName="payload-processing"
Nov 02 22:14:49.222164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.222140    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:14:49.243782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.243733    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m" podUID="1b3014cf-9f59-4bd5-9d00-d8963e680bee" containerName="payload-pre-processing" containerID="cri-o://7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30" gracePeriod=25
Nov 02 22:14:49.262187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.260535    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"]
Nov 02 22:14:49.296165 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.286354    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-dd28e6\""
Nov 02 22:14:49.296165 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.289363    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-dd28e6-dockercfg-xtr2b\""
Nov 02 22:14:49.392006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.372035    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11674de5-804a-4164-8838-f06115b70e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql\" (UID: \"11674de5-804a-4164-8838-f06115b70e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:14:49.392006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.372200    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fgs\" (UniqueName: \"kubernetes.io/projected/11674de5-804a-4164-8838-f06115b70e7f-kube-api-access-26fgs\") pod \"payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql\" (UID: \"11674de5-804a-4164-8838-f06115b70e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:14:49.424208 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.422740    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"]
Nov 02 22:14:49.428712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.428688    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:14:49.447857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.447018    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"]
Nov 02 22:14:49.473452 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.473411    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11674de5-804a-4164-8838-f06115b70e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql\" (UID: \"11674de5-804a-4164-8838-f06115b70e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:14:49.473652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.473495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fgs\" (UniqueName: \"kubernetes.io/projected/11674de5-804a-4164-8838-f06115b70e7f-kube-api-access-26fgs\") pod \"payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql\" (UID: \"11674de5-804a-4164-8838-f06115b70e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:14:49.473652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.473570    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh\" (UID: \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:14:49.473652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.473599    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74pkh\" (UniqueName: \"kubernetes.io/projected/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-kube-api-access-74pkh\") pod \"payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh\" (UID: \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:14:49.474242 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.474215    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11674de5-804a-4164-8838-f06115b70e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql\" (UID: \"11674de5-804a-4164-8838-f06115b70e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:14:49.530930 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.530898    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fgs\" (UniqueName: \"kubernetes.io/projected/11674de5-804a-4164-8838-f06115b70e7f-kube-api-access-26fgs\") pod \"payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql\" (UID: \"11674de5-804a-4164-8838-f06115b70e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:14:49.574930 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.574893    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh\" (UID: \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:14:49.574930 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.574942    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74pkh\" (UniqueName: \"kubernetes.io/projected/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-kube-api-access-74pkh\") pod \"payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh\" (UID: \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:14:49.575655 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.575636    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh\" (UID: \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:14:49.621967 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.621939    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:14:49.631855 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.631830    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74pkh\" (UniqueName: \"kubernetes.io/projected/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-kube-api-access-74pkh\") pod \"payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh\" (UID: \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:14:49.632939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.632921    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:14:49.675707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.675640    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68345e06-0f8b-4885-ab52-02a3271750b3-kube-api-access-nfbcr\" (UniqueName: \"kubernetes.io/projected/68345e06-0f8b-4885-ab52-02a3271750b3-kube-api-access-nfbcr\") pod \"68345e06-0f8b-4885-ab52-02a3271750b3\" (UID: \"68345e06-0f8b-4885-ab52-02a3271750b3\") "
Nov 02 22:14:49.675707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.675696    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68345e06-0f8b-4885-ab52-02a3271750b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68345e06-0f8b-4885-ab52-02a3271750b3-plugins-config-volume\") pod \"68345e06-0f8b-4885-ab52-02a3271750b3\" (UID: \"68345e06-0f8b-4885-ab52-02a3271750b3\") "
Nov 02 22:14:49.675892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.675719    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3014cf-9f59-4bd5-9d00-d8963e680bee-kube-api-access-wtcxd\" (UniqueName: \"kubernetes.io/projected/1b3014cf-9f59-4bd5-9d00-d8963e680bee-kube-api-access-wtcxd\") pod \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\" (UID: \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\") "
Nov 02 22:14:49.675892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.675778    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3014cf-9f59-4bd5-9d00-d8963e680bee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3014cf-9f59-4bd5-9d00-d8963e680bee-plugins-config-volume\") pod \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\" (UID: \"1b3014cf-9f59-4bd5-9d00-d8963e680bee\") "
Nov 02 22:14:49.676142 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.676085    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68345e06-0f8b-4885-ab52-02a3271750b3-plugins-config-volume" pod "68345e06-0f8b-4885-ab52-02a3271750b3" (UID: "68345e06-0f8b-4885-ab52-02a3271750b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:49.676247 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.676210    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3014cf-9f59-4bd5-9d00-d8963e680bee-plugins-config-volume" pod "1b3014cf-9f59-4bd5-9d00-d8963e680bee" (UID: "1b3014cf-9f59-4bd5-9d00-d8963e680bee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:14:49.677861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.677834    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3014cf-9f59-4bd5-9d00-d8963e680bee-kube-api-access-wtcxd" pod "1b3014cf-9f59-4bd5-9d00-d8963e680bee" (UID: "1b3014cf-9f59-4bd5-9d00-d8963e680bee"). InnerVolumeSpecName "kube-api-access-wtcxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:49.677861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.677853    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68345e06-0f8b-4885-ab52-02a3271750b3-kube-api-access-nfbcr" pod "68345e06-0f8b-4885-ab52-02a3271750b3" (UID: "68345e06-0f8b-4885-ab52-02a3271750b3"). InnerVolumeSpecName "kube-api-access-nfbcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:14:49.706168 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.706143    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:14:49.758304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.758260    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:14:49.777480 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.777444    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68345e06-0f8b-4885-ab52-02a3271750b3-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:49.777613 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.777486    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtcxd\" (UniqueName: \"kubernetes.io/projected/1b3014cf-9f59-4bd5-9d00-d8963e680bee-kube-api-access-wtcxd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:49.777613 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.777502    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3014cf-9f59-4bd5-9d00-d8963e680bee-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:49.777613 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:49.777517    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfbcr\" (UniqueName: \"kubernetes.io/projected/68345e06-0f8b-4885-ab52-02a3271750b3-kube-api-access-nfbcr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:14:50.097226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.097191    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"]
Nov 02 22:14:50.179445 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:14:50.179413    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1d0b84_23f8_4ffe_be30_9f71cbc95655.slice/crio-ca2196885994453733baceb1656a0f5fc5a65b6edf1457d8eeddaf0d206906a3 WatchSource:0}: Error finding container ca2196885994453733baceb1656a0f5fc5a65b6edf1457d8eeddaf0d206906a3: Status 404 returned error can't find the container with id ca2196885994453733baceb1656a0f5fc5a65b6edf1457d8eeddaf0d206906a3
Nov 02 22:14:50.182163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.182141    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"]
Nov 02 22:14:50.306626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.306587    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh" event={"ID":"2f1d0b84-23f8-4ffe-be30-9f71cbc95655","Type":"ContainerStarted","Data":"ca2196885994453733baceb1656a0f5fc5a65b6edf1457d8eeddaf0d206906a3"}
Nov 02 22:14:50.309032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.309000    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql" event={"ID":"11674de5-804a-4164-8838-f06115b70e7f","Type":"ContainerStarted","Data":"b378d96dafe50ca38b99feec624f672ac418f9bc106e4d35988b80cb8d991542"}
Nov 02 22:14:50.309223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.309054    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql" event={"ID":"11674de5-804a-4164-8838-f06115b70e7f","Type":"ContainerStarted","Data":"08f8a680d726434f7e6466e941463ade0cf8226a29c98a70ee162a60c60f220b"}
Nov 02 22:14:50.310002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.309979    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:14:50.311575 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.311552    4528 generic.go:356] "Generic (PLEG): container finished" podID="68345e06-0f8b-4885-ab52-02a3271750b3" containerID="daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b" exitCode=0
Nov 02 22:14:50.311757 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.311617    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"
Nov 02 22:14:50.312450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.311633    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8" event={"ID":"68345e06-0f8b-4885-ab52-02a3271750b3","Type":"ContainerDied","Data":"daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b"}
Nov 02 22:14:50.312623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.312604    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8" event={"ID":"68345e06-0f8b-4885-ab52-02a3271750b3","Type":"ContainerDied","Data":"99dc3fca7393b546cc3452695b91c3431a9a7caab37b7434060155437a7a3348"}
Nov 02 22:14:50.312744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.312732    4528 scope.go:122] "RemoveContainer" containerID="daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b"
Nov 02 22:14:50.316961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.316924    4528 generic.go:356] "Generic (PLEG): container finished" podID="1b3014cf-9f59-4bd5-9d00-d8963e680bee" containerID="7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30" exitCode=0
Nov 02 22:14:50.317272 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.317252    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m" event={"ID":"1b3014cf-9f59-4bd5-9d00-d8963e680bee","Type":"ContainerDied","Data":"7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30"}
Nov 02 22:14:50.317414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.317400    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m" event={"ID":"1b3014cf-9f59-4bd5-9d00-d8963e680bee","Type":"ContainerDied","Data":"34644f5caa05c286565e64c78e3c1d9fbdde6dbbb9b740ff81d1e5bb14d8e57f"}
Nov 02 22:14:50.317610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.317577    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"
Nov 02 22:14:50.356435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.356415    4528 scope.go:122] "RemoveContainer" containerID="daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b"
Nov 02 22:14:50.356735 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:14:50.356707    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b\": container with ID starting with daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b not found: ID does not exist" containerID="daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b"
Nov 02 22:14:50.356775 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.356746    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b"} err="failed to get container status \"daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b\": rpc error: code = NotFound desc = could not find container \"daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b\": container with ID starting with daae8aecc69012f45778e3361847cb6ba1af6e112274fdb554c99307a0317d0b not found: ID does not exist"
Nov 02 22:14:50.356775 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.356763    4528 scope.go:122] "RemoveContainer" containerID="7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30"
Nov 02 22:14:50.367063 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.367047    4528 scope.go:122] "RemoveContainer" containerID="7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30"
Nov 02 22:14:50.367375 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:14:50.367357    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30\": container with ID starting with 7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30 not found: ID does not exist" containerID="7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30"
Nov 02 22:14:50.367448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.367382    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30"} err="failed to get container status \"7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30\": rpc error: code = NotFound desc = could not find container \"7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30\": container with ID starting with 7081f5e7ad1d4ed706b3143b199effc9d690deadd966fcf16629cd4bb93c9a30 not found: ID does not exist"
Nov 02 22:14:50.431363 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.431307    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql" podStartSLOduration=1.431292091 podStartE2EDuration="1.431292091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:50.427676135 +0000 UTC m=+3977.691212454" watchObservedRunningTime="2026-11-02 22:14:50.431292091 +0000 UTC m=+3977.694828442"
Nov 02 22:14:50.565320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.565239    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"]
Nov 02 22:14:50.574145 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.572155    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4585ee-7bfb744db7-mnq8m"]
Nov 02 22:14:50.657578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.657484    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"]
Nov 02 22:14:50.662675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.662648    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4585ee-66b87555df-klwb8"]
Nov 02 22:14:50.933287 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.933199    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3014cf-9f59-4bd5-9d00-d8963e680bee" path="/var/lib/kubelet/pods/1b3014cf-9f59-4bd5-9d00-d8963e680bee/volumes"
Nov 02 22:14:50.933712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:50.933688    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68345e06-0f8b-4885-ab52-02a3271750b3" path="/var/lib/kubelet/pods/68345e06-0f8b-4885-ab52-02a3271750b3/volumes"
Nov 02 22:14:51.334874 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:51.334830    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh" event={"ID":"2f1d0b84-23f8-4ffe-be30-9f71cbc95655","Type":"ContainerStarted","Data":"0ea385dd535228c729c2e717f338217de1c905ce1a76b3822853f49a40767e2b"}
Nov 02 22:14:51.450368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:51.450317    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh" podStartSLOduration=2.450303224 podStartE2EDuration="2.450303224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:51.447344798 +0000 UTC m=+3978.710881115" watchObservedRunningTime="2026-11-02 22:14:51.450303224 +0000 UTC m=+3978.713839559"
Nov 02 22:14:52.341488 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:52.341450    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:14:59.396144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:59.396078    4528 generic.go:356] "Generic (PLEG): container finished" podID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerID="52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538" exitCode=0
Nov 02 22:14:59.396575 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:59.396226    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" event={"ID":"313de020-6c5e-455c-b7a0-d1a15228e8a1","Type":"ContainerDied","Data":"52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538"}
Nov 02 22:14:59.396729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:59.396715    4528 scope.go:122] "RemoveContainer" containerID="52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538"
Nov 02 22:14:59.398405 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:59.398378    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:14:59.398464 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:59.398443    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:14:59.398512 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:14:59.398476    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:15:00.171789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.171737    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"]
Nov 02 22:15:00.172501 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.172480    4528 cpu_manager.go:414] "containerMap: removing container" podUID="68345e06-0f8b-4885-ab52-02a3271750b3" containerName="payload-processing"
Nov 02 22:15:00.172501 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.172500    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68345e06-0f8b-4885-ab52-02a3271750b3" containerName="payload-processing"
Nov 02 22:15:00.172691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.172521    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1b3014cf-9f59-4bd5-9d00-d8963e680bee" containerName="payload-pre-processing"
Nov 02 22:15:00.172691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.172527    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3014cf-9f59-4bd5-9d00-d8963e680bee" containerName="payload-pre-processing"
Nov 02 22:15:00.172691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.172627    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="68345e06-0f8b-4885-ab52-02a3271750b3" containerName="payload-processing"
Nov 02 22:15:00.172691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.172637    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3014cf-9f59-4bd5-9d00-d8963e680bee" containerName="payload-pre-processing"
Nov 02 22:15:00.176218 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.176197    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"]
Nov 02 22:15:00.176364 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.176327    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:00.180390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.180371    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"]
Nov 02 22:15:00.180522 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.180505    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:00.184296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.184276    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:00.222301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.222278    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"]
Nov 02 22:15:00.226534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.226510    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"]
Nov 02 22:15:00.226685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.226634    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:00.230764 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.230741    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"]
Nov 02 22:15:00.230908 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.230870    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:00.235369 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.235343    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"]
Nov 02 22:15:00.235493 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.235452    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:00.240250 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.240227    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"]
Nov 02 22:15:00.240374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.240338    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:00.244267 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.244246    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"]
Nov 02 22:15:00.244381 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.244366    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:00.250589 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.250568    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"]
Nov 02 22:15:00.250855 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.250690    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:00.253845 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.253822    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7fd\" (UniqueName: \"kubernetes.io/projected/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-kube-api-access-vj7fd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g\" (UID: \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:00.253951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.253871    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zq5w\" (UniqueName: \"kubernetes.io/projected/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-kube-api-access-6zq5w\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh\" (UID: \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:00.254002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.253966    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh\" (UID: \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:00.254002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.253991    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g\" (UID: \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:00.254088 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.254036    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chm2\" (UniqueName: \"kubernetes.io/projected/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-kube-api-access-8chm2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl\" (UID: \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:00.254153 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.254091    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl\" (UID: \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:00.255750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.255733    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"]
Nov 02 22:15:00.255876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.255857    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:00.262633 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.262613    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:00.270372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.269150    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:15:00.270372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.269249    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:15:00.270622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.270597    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:15:00.280530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.274617    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"]
Nov 02 22:15:00.282779 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.281251    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"]
Nov 02 22:15:00.282779 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.281355    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:00.292133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.285203    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"]
Nov 02 22:15:00.292133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.288895    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"]
Nov 02 22:15:00.310758 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.307200    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"]
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358297    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cda15-3a4d-4450-b922-aa263f332786-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n\" (UID: \"3e6cda15-3a4d-4450-b922-aa263f332786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358379    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7fd\" (UniqueName: \"kubernetes.io/projected/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-kube-api-access-vj7fd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g\" (UID: \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358420    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be619aa4-7336-40a9-b5d4-1a5ba5d69129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4\" (UID: \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358478    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qb64\" (UniqueName: \"kubernetes.io/projected/507f7d40-be0c-4471-9821-04f1fb7480e1-kube-api-access-9qb64\") pod \"maas-api-key-cleanup-29894295-7mprd\" (UID: \"507f7d40-be0c-4471-9821-04f1fb7480e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358522    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbvt\" (UniqueName: \"kubernetes.io/projected/be619aa4-7336-40a9-b5d4-1a5ba5d69129-kube-api-access-jqbvt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4\" (UID: \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358559    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zq5w\" (UniqueName: \"kubernetes.io/projected/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-kube-api-access-6zq5w\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh\" (UID: \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358595    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330703a-25dd-4c73-a5b8-296354013803-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22\" (UID: \"9330703a-25dd-4c73-a5b8-296354013803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358658    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21ffde3-3d62-4a69-abfc-3fecce92eb86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6\" (UID: \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358702    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh\" (UID: \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358740    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29r5v\" (UniqueName: \"kubernetes.io/projected/9330703a-25dd-4c73-a5b8-296354013803-kube-api-access-29r5v\") pod \"maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22\" (UID: \"9330703a-25dd-4c73-a5b8-296354013803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358785    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g\" (UID: \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358814    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p\" (UID: \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358850    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnh25\" (UniqueName: \"kubernetes.io/projected/3e6cda15-3a4d-4450-b922-aa263f332786-kube-api-access-gnh25\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n\" (UID: \"3e6cda15-3a4d-4450-b922-aa263f332786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358887    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rj5\" (UniqueName: \"kubernetes.io/projected/a21ffde3-3d62-4a69-abfc-3fecce92eb86-kube-api-access-45rj5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6\" (UID: \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358937    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660176c3-ce07-49f1-af67-55c1d191a517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv\" (UID: \"660176c3-ce07-49f1-af67-55c1d191a517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.358976    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chm2\" (UniqueName: \"kubernetes.io/projected/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-kube-api-access-8chm2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl\" (UID: \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.359020    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk286\" (UniqueName: \"kubernetes.io/projected/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-kube-api-access-tk286\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g\" (UID: \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:00.359132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.359071    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl\" (UID: \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:00.360355 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.359156    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f7d40-be0c-4471-9821-04f1fb7480e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-7mprd\" (UID: \"507f7d40-be0c-4471-9821-04f1fb7480e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:00.360355 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.359194    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc0a56-99f7-4467-8501-8a50b683cb6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4\" (UID: \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:00.360355 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.359297    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g\" (UID: \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:00.360355 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.359330    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvn4k\" (UniqueName: \"kubernetes.io/projected/660176c3-ce07-49f1-af67-55c1d191a517-kube-api-access-dvn4k\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv\" (UID: \"660176c3-ce07-49f1-af67-55c1d191a517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:00.360355 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.359442    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwr9g\" (UniqueName: \"kubernetes.io/projected/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-kube-api-access-xwr9g\") pod \"maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p\" (UID: \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:00.360355 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.359483    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gxd\" (UniqueName: \"kubernetes.io/projected/0dbc0a56-99f7-4467-8501-8a50b683cb6e-kube-api-access-49gxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4\" (UID: \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:00.363227 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.360804    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh\" (UID: \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:00.363227 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.360989    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl\" (UID: \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:00.363227 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.361565    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g\" (UID: \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:00.374948 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.374913    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"]
Nov 02 22:15:00.375450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.375429    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"]
Nov 02 22:15:00.382534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.382070    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"]
Nov 02 22:15:00.385515 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.385484    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"]
Nov 02 22:15:00.434391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.430658    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" event={"ID":"313de020-6c5e-455c-b7a0-d1a15228e8a1","Type":"ContainerStarted","Data":"154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be"}
Nov 02 22:15:00.443173 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.440060    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"]
Nov 02 22:15:00.445372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.444442    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"]
Nov 02 22:15:00.451005 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.450859    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zq5w\" (UniqueName: \"kubernetes.io/projected/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-kube-api-access-6zq5w\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh\" (UID: \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:00.451391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.451042    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chm2\" (UniqueName: \"kubernetes.io/projected/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-kube-api-access-8chm2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl\" (UID: \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:00.452273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.452221    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7fd\" (UniqueName: \"kubernetes.io/projected/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-kube-api-access-vj7fd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g\" (UID: \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:00.453479 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.453400    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"]
Nov 02 22:15:00.458740 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.458688    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"]
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460447    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk286\" (UniqueName: \"kubernetes.io/projected/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-kube-api-access-tk286\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g\" (UID: \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460500    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f7d40-be0c-4471-9821-04f1fb7480e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-7mprd\" (UID: \"507f7d40-be0c-4471-9821-04f1fb7480e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460519    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc0a56-99f7-4467-8501-8a50b683cb6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4\" (UID: \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460576    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g\" (UID: \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460592    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvn4k\" (UniqueName: \"kubernetes.io/projected/660176c3-ce07-49f1-af67-55c1d191a517-kube-api-access-dvn4k\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv\" (UID: \"660176c3-ce07-49f1-af67-55c1d191a517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460642    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwr9g\" (UniqueName: \"kubernetes.io/projected/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-kube-api-access-xwr9g\") pod \"maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p\" (UID: \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460660    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gxd\" (UniqueName: \"kubernetes.io/projected/0dbc0a56-99f7-4467-8501-8a50b683cb6e-kube-api-access-49gxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4\" (UID: \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460685    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cda15-3a4d-4450-b922-aa263f332786-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n\" (UID: \"3e6cda15-3a4d-4450-b922-aa263f332786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460731    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be619aa4-7336-40a9-b5d4-1a5ba5d69129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4\" (UID: \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460763    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qb64\" (UniqueName: \"kubernetes.io/projected/507f7d40-be0c-4471-9821-04f1fb7480e1-kube-api-access-9qb64\") pod \"maas-api-key-cleanup-29894295-7mprd\" (UID: \"507f7d40-be0c-4471-9821-04f1fb7480e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460784    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbvt\" (UniqueName: \"kubernetes.io/projected/be619aa4-7336-40a9-b5d4-1a5ba5d69129-kube-api-access-jqbvt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4\" (UID: \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460816    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330703a-25dd-4c73-a5b8-296354013803-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22\" (UID: \"9330703a-25dd-4c73-a5b8-296354013803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460847    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21ffde3-3d62-4a69-abfc-3fecce92eb86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6\" (UID: \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460870    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29r5v\" (UniqueName: \"kubernetes.io/projected/9330703a-25dd-4c73-a5b8-296354013803-kube-api-access-29r5v\") pod \"maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22\" (UID: \"9330703a-25dd-4c73-a5b8-296354013803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460894    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p\" (UID: \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460915    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnh25\" (UniqueName: \"kubernetes.io/projected/3e6cda15-3a4d-4450-b922-aa263f332786-kube-api-access-gnh25\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n\" (UID: \"3e6cda15-3a4d-4450-b922-aa263f332786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460934    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rj5\" (UniqueName: \"kubernetes.io/projected/a21ffde3-3d62-4a69-abfc-3fecce92eb86-kube-api-access-45rj5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6\" (UID: \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.460970    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660176c3-ce07-49f1-af67-55c1d191a517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv\" (UID: \"660176c3-ce07-49f1-af67-55c1d191a517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.461521    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660176c3-ce07-49f1-af67-55c1d191a517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv\" (UID: \"660176c3-ce07-49f1-af67-55c1d191a517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.462531    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be619aa4-7336-40a9-b5d4-1a5ba5d69129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4\" (UID: \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.462920    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc0a56-99f7-4467-8501-8a50b683cb6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4\" (UID: \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.463546    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g\" (UID: \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:00.464390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.464347    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21ffde3-3d62-4a69-abfc-3fecce92eb86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6\" (UID: \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:00.465675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.464528    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cda15-3a4d-4450-b922-aa263f332786-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n\" (UID: \"3e6cda15-3a4d-4450-b922-aa263f332786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:00.465675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.464600    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f7d40-be0c-4471-9821-04f1fb7480e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-7mprd\" (UID: \"507f7d40-be0c-4471-9821-04f1fb7480e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:00.465675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.465062    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p\" (UID: \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:00.465675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.465478    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330703a-25dd-4c73-a5b8-296354013803-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22\" (UID: \"9330703a-25dd-4c73-a5b8-296354013803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:00.493843 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.493810    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:00.501692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.501442    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:00.532150 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.528026    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvn4k\" (UniqueName: \"kubernetes.io/projected/660176c3-ce07-49f1-af67-55c1d191a517-kube-api-access-dvn4k\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv\" (UID: \"660176c3-ce07-49f1-af67-55c1d191a517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:00.532609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.532563    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:00.537792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.536422    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwr9g\" (UniqueName: \"kubernetes.io/projected/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-kube-api-access-xwr9g\") pod \"maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p\" (UID: \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:00.537792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.536627    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29r5v\" (UniqueName: \"kubernetes.io/projected/9330703a-25dd-4c73-a5b8-296354013803-kube-api-access-29r5v\") pod \"maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22\" (UID: \"9330703a-25dd-4c73-a5b8-296354013803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:00.537792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.536996    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnh25\" (UniqueName: \"kubernetes.io/projected/3e6cda15-3a4d-4450-b922-aa263f332786-kube-api-access-gnh25\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n\" (UID: \"3e6cda15-3a4d-4450-b922-aa263f332786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:00.537792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.537744    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gxd\" (UniqueName: \"kubernetes.io/projected/0dbc0a56-99f7-4467-8501-8a50b683cb6e-kube-api-access-49gxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4\" (UID: \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:00.538130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.537846    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qb64\" (UniqueName: \"kubernetes.io/projected/507f7d40-be0c-4471-9821-04f1fb7480e1-kube-api-access-9qb64\") pod \"maas-api-key-cleanup-29894295-7mprd\" (UID: \"507f7d40-be0c-4471-9821-04f1fb7480e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:00.538130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.538088    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk286\" (UniqueName: \"kubernetes.io/projected/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-kube-api-access-tk286\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g\" (UID: \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:00.538420 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.538398    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rj5\" (UniqueName: \"kubernetes.io/projected/a21ffde3-3d62-4a69-abfc-3fecce92eb86-kube-api-access-45rj5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6\" (UID: \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:00.539146 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.539092    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbvt\" (UniqueName: \"kubernetes.io/projected/be619aa4-7336-40a9-b5d4-1a5ba5d69129-kube-api-access-jqbvt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4\" (UID: \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:00.547040 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.547016    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:00.555290 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.555263    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:00.564703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.564674    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:00.573763 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.573741    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:00.592308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.591883    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:00.641792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.641544    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:00.667771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.667410    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:00.791694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.791557    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:00.793982 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:00.793896    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:01.138009 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.137855    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"]
Nov 02 22:15:01.191789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.191725    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:15:01.393311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.393278    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"]
Nov 02 22:15:01.433534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.411947    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:15:01.433534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.430289    4528 patch_prober.go:29] interesting pod/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 22:15:01.433534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.430347    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 22:15:01.449582 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.448967    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"]
Nov 02 22:15:01.451178 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.450799    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" event={"ID":"e5a4167b-2636-40ef-842d-9ea3cc8c46ea","Type":"ContainerStarted","Data":"11ae6eb943eda3647b1c2d1df85252cded538e72a9a81ffaef6ed08d02a074a7"}
Nov 02 22:15:01.454719 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.454691    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"]
Nov 02 22:15:01.498785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.497475    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"]
Nov 02 22:15:01.533136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.519198    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"]
Nov 02 22:15:01.575789 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:15:01.575679    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21ffde3_3d62_4a69_abfc_3fecce92eb86.slice/crio-7142c8fc1cab5823bf208e9565839f0b9bbb041443e035fcbd73c7f946c00f8c WatchSource:0}: Error finding container 7142c8fc1cab5823bf208e9565839f0b9bbb041443e035fcbd73c7f946c00f8c: Status 404 returned error can't find the container with id 7142c8fc1cab5823bf208e9565839f0b9bbb041443e035fcbd73c7f946c00f8c
Nov 02 22:15:01.578895 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:15:01.578868    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660176c3_ce07_49f1_af67_55c1d191a517.slice/crio-77f06014bfd653bb19736e1d044b0eba2362ef0c69ba938d2e8f309b1579da2b WatchSource:0}: Error finding container 77f06014bfd653bb19736e1d044b0eba2362ef0c69ba938d2e8f309b1579da2b: Status 404 returned error can't find the container with id 77f06014bfd653bb19736e1d044b0eba2362ef0c69ba938d2e8f309b1579da2b
Nov 02 22:15:01.579033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.578924    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"]
Nov 02 22:15:01.583886 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.582915    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"]
Nov 02 22:15:01.593269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.593244    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"]
Nov 02 22:15:01.597022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.596993    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"]
Nov 02 22:15:01.635933 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:15:01.635897    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507f7d40_be0c_4471_9821_04f1fb7480e1.slice/crio-04566f4fc6281eedc0d556afec1686826c9f66f5d4dd6e6b9ad325313a692a2e WatchSource:0}: Error finding container 04566f4fc6281eedc0d556afec1686826c9f66f5d4dd6e6b9ad325313a692a2e: Status 404 returned error can't find the container with id 04566f4fc6281eedc0d556afec1686826c9f66f5d4dd6e6b9ad325313a692a2e
Nov 02 22:15:01.671990 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.671329    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"]
Nov 02 22:15:01.696012 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:01.695939    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"]
Nov 02 22:15:02.345603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.345350    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:15:02.439060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.438947    4528 patch_prober.go:29] interesting pod/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 22:15:02.439060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.439004    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 22:15:02.463506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.463467    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" event={"ID":"b736b5d4-392b-4ee7-8757-dfe9b0b63c16","Type":"ContainerStarted","Data":"a2f5454e1aef41b5d9048846e3c563563a592e222e3bae27c1c8486599f3ce83"}
Nov 02 22:15:02.464223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.463534    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" event={"ID":"b736b5d4-392b-4ee7-8757-dfe9b0b63c16","Type":"ContainerStarted","Data":"4629bb4166a9369e5c5ea031e27cfde4e9091ea954fdd0c8bdfdba35700fd244"}
Nov 02 22:15:02.466577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.465471    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" event={"ID":"be619aa4-7336-40a9-b5d4-1a5ba5d69129","Type":"ContainerStarted","Data":"64fee73eff5d241fb08c42fb67292867c4fd1668ce1c20ca9b823447ffa0713f"}
Nov 02 22:15:02.466577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.465512    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" event={"ID":"be619aa4-7336-40a9-b5d4-1a5ba5d69129","Type":"ContainerStarted","Data":"d326a5ca97f9f08eee83f6001b2c9b0b7c395688f5770eb3393833eb4a646e4e"}
Nov 02 22:15:02.468080 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.467746    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" event={"ID":"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1","Type":"ContainerStarted","Data":"99b158de20ab07936e58acd5a2bcbf23fb95566febe2be0719e7e6671f236eb0"}
Nov 02 22:15:02.468080 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.467817    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" event={"ID":"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1","Type":"ContainerStarted","Data":"73f6189ce23cfaad8693e77b3747bb6ef26d7842076c232fe348203a85c4aff2"}
Nov 02 22:15:02.470815 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.470187    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" event={"ID":"0dbc0a56-99f7-4467-8501-8a50b683cb6e","Type":"ContainerStarted","Data":"579f78bcdf5c5446ecfd08243a42a5aee2ae99bb876033c30c9e59c039adc0ce"}
Nov 02 22:15:02.470815 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.470232    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" event={"ID":"0dbc0a56-99f7-4467-8501-8a50b683cb6e","Type":"ContainerStarted","Data":"280a47280d648ba4b0a6b43cf3a92299f55e503808c88ca11e88dd91217cdddd"}
Nov 02 22:15:02.473190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.472564    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" event={"ID":"507f7d40-be0c-4471-9821-04f1fb7480e1","Type":"ContainerStarted","Data":"bea48d7653bee747f88517aaa112f416f0bdca2037401647c58878202a7a20e0"}
Nov 02 22:15:02.473190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.472615    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" event={"ID":"507f7d40-be0c-4471-9821-04f1fb7480e1","Type":"ContainerStarted","Data":"04566f4fc6281eedc0d556afec1686826c9f66f5d4dd6e6b9ad325313a692a2e"}
Nov 02 22:15:02.475261 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.474671    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" event={"ID":"e5a4167b-2636-40ef-842d-9ea3cc8c46ea","Type":"ContainerStarted","Data":"1e2c0202b7be13ad1e1838c9b5e9f706194d78d3496b6e4636a8572337e4fc2f"}
Nov 02 22:15:02.477310 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.477283    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" event={"ID":"9330703a-25dd-4c73-a5b8-296354013803","Type":"ContainerStarted","Data":"584d1bbd7b9c49277df72b9108fddf1f771918de72f6c058794616c238fb24ef"}
Nov 02 22:15:02.477793 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.477768    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" event={"ID":"9330703a-25dd-4c73-a5b8-296354013803","Type":"ContainerStarted","Data":"32fc11a91ddb1c886e35b48bdb13727dfa9512a8b451ab7394139147dcb54ee4"}
Nov 02 22:15:02.479256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.479232    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" event={"ID":"3e6cda15-3a4d-4450-b922-aa263f332786","Type":"ContainerStarted","Data":"5186e913b1c9a32bbc996ad5f22ef23ca0fc0a2217c7a5171e22ca9a3350aa7d"}
Nov 02 22:15:02.479866 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.479846    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" event={"ID":"3e6cda15-3a4d-4450-b922-aa263f332786","Type":"ContainerStarted","Data":"17cb5a25a5712b6eee060da9849838e007d8818265f289253e12421630b43bd1"}
Nov 02 22:15:02.481148 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.481080    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" event={"ID":"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f","Type":"ContainerStarted","Data":"4ad9937b6d2191af65711c9ad6cfe219083dd652084a9cc5b47769860c8bb6ce"}
Nov 02 22:15:02.483245 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.483216    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" event={"ID":"204147df-3fd6-4c5c-b2b4-691e5c45d9bc","Type":"ContainerStarted","Data":"9bbb8ac906e7e14d81a1831aa4ac6aa329de67369c5111d6a89b9695fcb98800"}
Nov 02 22:15:02.483340 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.483268    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" event={"ID":"204147df-3fd6-4c5c-b2b4-691e5c45d9bc","Type":"ContainerStarted","Data":"bb03d93ff566f85addd2d6bae6ff5d3faa6a1fadb4194a9c192cdc25ec96d05c"}
Nov 02 22:15:02.485015 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.484985    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" event={"ID":"a21ffde3-3d62-4a69-abfc-3fecce92eb86","Type":"ContainerStarted","Data":"aa2ff56e8abdbe86d070f0ccd4b367a3f0eeb40183c2cef94806674d4836bb07"}
Nov 02 22:15:02.485151 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.485018    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" event={"ID":"a21ffde3-3d62-4a69-abfc-3fecce92eb86","Type":"ContainerStarted","Data":"7142c8fc1cab5823bf208e9565839f0b9bbb041443e035fcbd73c7f946c00f8c"}
Nov 02 22:15:02.487710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.487041    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" event={"ID":"660176c3-ce07-49f1-af67-55c1d191a517","Type":"ContainerStarted","Data":"d35d17579375b1057f27b9b963e5c4efc6a23e6529a98875e488e210c3a7884d"}
Nov 02 22:15:02.487710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.487086    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" event={"ID":"660176c3-ce07-49f1-af67-55c1d191a517","Type":"ContainerStarted","Data":"77f06014bfd653bb19736e1d044b0eba2362ef0c69ba938d2e8f309b1579da2b"}
Nov 02 22:15:02.584901 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.584819    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" podStartSLOduration=2.584801112 podStartE2EDuration="2.584801112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:02.579193445 +0000 UTC m=+3989.842729762" watchObservedRunningTime="2026-11-02 22:15:02.584801112 +0000 UTC m=+3989.848337430"
Nov 02 22:15:02.696191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.696140    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" podStartSLOduration=2.696124923 podStartE2EDuration="2.696124923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:02.692206639 +0000 UTC m=+3989.955742957" watchObservedRunningTime="2026-11-02 22:15:02.696124923 +0000 UTC m=+3989.959661233"
Nov 02 22:15:02.837550 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.833571    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" podStartSLOduration=2.8335489579999997 podStartE2EDuration="2.833548958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:02.821294162 +0000 UTC m=+3990.084830482" watchObservedRunningTime="2026-11-02 22:15:02.833548958 +0000 UTC m=+3990.097085277"
Nov 02 22:15:02.937764 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:02.937671    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" podStartSLOduration=2.937649873 podStartE2EDuration="2.937649873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:02.935954173 +0000 UTC m=+3990.199490493" watchObservedRunningTime="2026-11-02 22:15:02.937649873 +0000 UTC m=+3990.201186191"
Nov 02 22:15:03.048452 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.048221    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" podStartSLOduration=3.048206239 podStartE2EDuration="3.048206239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.047351082 +0000 UTC m=+3990.310887402" watchObservedRunningTime="2026-11-02 22:15:03.048206239 +0000 UTC m=+3990.311742557"
Nov 02 22:15:03.055028 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.055003    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"]
Nov 02 22:15:03.058828 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.058807    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad1d0fb-8ctpk"]
Nov 02 22:15:03.161541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.161478    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" podStartSLOduration=3.161464287 podStartE2EDuration="3.161464287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.159941255 +0000 UTC m=+3990.423477573" watchObservedRunningTime="2026-11-02 22:15:03.161464287 +0000 UTC m=+3990.425000605"
Nov 02 22:15:03.350912 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.350887    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:15:03.393973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.393868    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" podStartSLOduration=3.393854845 podStartE2EDuration="3.393854845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.2817227 +0000 UTC m=+3990.545259031" watchObservedRunningTime="2026-11-02 22:15:03.393854845 +0000 UTC m=+3990.657391208"
Nov 02 22:15:03.395538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.395480    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" podStartSLOduration=3.395467546 podStartE2EDuration="3.395467546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.392819647 +0000 UTC m=+3990.656355963" watchObservedRunningTime="2026-11-02 22:15:03.395467546 +0000 UTC m=+3990.659003900"
Nov 02 22:15:03.422646 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.422524    4528 patch_prober.go:29] interesting pod/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 22:15:03.422646 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.422585    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 22:15:03.496402 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.496233    4528 generic.go:356] "Generic (PLEG): container finished" podID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerID="bea48d7653bee747f88517aaa112f416f0bdca2037401647c58878202a7a20e0" exitCode=60
Nov 02 22:15:03.496402 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.496317    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" event={"ID":"507f7d40-be0c-4471-9821-04f1fb7480e1","Type":"ContainerDied","Data":"bea48d7653bee747f88517aaa112f416f0bdca2037401647c58878202a7a20e0"}
Nov 02 22:15:03.497222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.497204    4528 scope.go:122] "RemoveContainer" containerID="bea48d7653bee747f88517aaa112f416f0bdca2037401647c58878202a7a20e0"
Nov 02 22:15:03.498090 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.498066    4528 generic.go:356] "Generic (PLEG): container finished" podID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerID="1e2c0202b7be13ad1e1838c9b5e9f706194d78d3496b6e4636a8572337e4fc2f" exitCode=60
Nov 02 22:15:03.498228 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.498204    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" event={"ID":"e5a4167b-2636-40ef-842d-9ea3cc8c46ea","Type":"ContainerDied","Data":"1e2c0202b7be13ad1e1838c9b5e9f706194d78d3496b6e4636a8572337e4fc2f"}
Nov 02 22:15:03.498511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.498493    4528 scope.go:122] "RemoveContainer" containerID="1e2c0202b7be13ad1e1838c9b5e9f706194d78d3496b6e4636a8572337e4fc2f"
Nov 02 22:15:03.500879 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.500855    4528 generic.go:356] "Generic (PLEG): container finished" podID="9330703a-25dd-4c73-a5b8-296354013803" containerID="584d1bbd7b9c49277df72b9108fddf1f771918de72f6c058794616c238fb24ef" exitCode=60
Nov 02 22:15:03.501007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.500940    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" event={"ID":"9330703a-25dd-4c73-a5b8-296354013803","Type":"ContainerDied","Data":"584d1bbd7b9c49277df72b9108fddf1f771918de72f6c058794616c238fb24ef"}
Nov 02 22:15:03.527434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.527415    4528 scope.go:122] "RemoveContainer" containerID="584d1bbd7b9c49277df72b9108fddf1f771918de72f6c058794616c238fb24ef"
Nov 02 22:15:03.531255 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.529699    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" podStartSLOduration=3.529682264 podStartE2EDuration="3.529682264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.527315143 +0000 UTC m=+3990.790851462" watchObservedRunningTime="2026-11-02 22:15:03.529682264 +0000 UTC m=+3990.793218585"
Nov 02 22:15:03.531255 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.530681    4528 generic.go:356] "Generic (PLEG): container finished" podID="3e6cda15-3a4d-4450-b922-aa263f332786" containerID="5186e913b1c9a32bbc996ad5f22ef23ca0fc0a2217c7a5171e22ca9a3350aa7d" exitCode=60
Nov 02 22:15:03.531255 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.530749    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" event={"ID":"3e6cda15-3a4d-4450-b922-aa263f332786","Type":"ContainerDied","Data":"5186e913b1c9a32bbc996ad5f22ef23ca0fc0a2217c7a5171e22ca9a3350aa7d"}
Nov 02 22:15:03.531255 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.531076    4528 scope.go:122] "RemoveContainer" containerID="5186e913b1c9a32bbc996ad5f22ef23ca0fc0a2217c7a5171e22ca9a3350aa7d"
Nov 02 22:15:03.534210 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.534166    4528 generic.go:356] "Generic (PLEG): container finished" podID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerID="25e4755adf89cd0166cf957c9c32a5b4ac2a8253d495b74043d67e4ab584cf51" exitCode=60
Nov 02 22:15:03.534407 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.534261    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" event={"ID":"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f","Type":"ContainerDied","Data":"25e4755adf89cd0166cf957c9c32a5b4ac2a8253d495b74043d67e4ab584cf51"}
Nov 02 22:15:03.534693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.534629    4528 scope.go:122] "RemoveContainer" containerID="25e4755adf89cd0166cf957c9c32a5b4ac2a8253d495b74043d67e4ab584cf51"
Nov 02 22:15:03.536854 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.536833    4528 generic.go:356] "Generic (PLEG): container finished" podID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerID="9bbb8ac906e7e14d81a1831aa4ac6aa329de67369c5111d6a89b9695fcb98800" exitCode=60
Nov 02 22:15:03.536987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.536927    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" event={"ID":"204147df-3fd6-4c5c-b2b4-691e5c45d9bc","Type":"ContainerDied","Data":"9bbb8ac906e7e14d81a1831aa4ac6aa329de67369c5111d6a89b9695fcb98800"}
Nov 02 22:15:03.537303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.537272    4528 scope.go:122] "RemoveContainer" containerID="9bbb8ac906e7e14d81a1831aa4ac6aa329de67369c5111d6a89b9695fcb98800"
Nov 02 22:15:03.538943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.538922    4528 generic.go:356] "Generic (PLEG): container finished" podID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerID="aa2ff56e8abdbe86d070f0ccd4b367a3f0eeb40183c2cef94806674d4836bb07" exitCode=60
Nov 02 22:15:03.539020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.539001    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" event={"ID":"a21ffde3-3d62-4a69-abfc-3fecce92eb86","Type":"ContainerDied","Data":"aa2ff56e8abdbe86d070f0ccd4b367a3f0eeb40183c2cef94806674d4836bb07"}
Nov 02 22:15:03.539308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.539292    4528 scope.go:122] "RemoveContainer" containerID="aa2ff56e8abdbe86d070f0ccd4b367a3f0eeb40183c2cef94806674d4836bb07"
Nov 02 22:15:03.545389 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.545370    4528 generic.go:356] "Generic (PLEG): container finished" podID="660176c3-ce07-49f1-af67-55c1d191a517" containerID="d35d17579375b1057f27b9b963e5c4efc6a23e6529a98875e488e210c3a7884d" exitCode=60
Nov 02 22:15:03.545520 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.545504    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" event={"ID":"660176c3-ce07-49f1-af67-55c1d191a517","Type":"ContainerDied","Data":"d35d17579375b1057f27b9b963e5c4efc6a23e6529a98875e488e210c3a7884d"}
Nov 02 22:15:03.547307 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.547287    4528 generic.go:356] "Generic (PLEG): container finished" podID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerID="a2f5454e1aef41b5d9048846e3c563563a592e222e3bae27c1c8486599f3ce83" exitCode=60
Nov 02 22:15:03.547387 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.547378    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" event={"ID":"b736b5d4-392b-4ee7-8757-dfe9b0b63c16","Type":"ContainerDied","Data":"a2f5454e1aef41b5d9048846e3c563563a592e222e3bae27c1c8486599f3ce83"}
Nov 02 22:15:03.547643 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.547624    4528 scope.go:122] "RemoveContainer" containerID="a2f5454e1aef41b5d9048846e3c563563a592e222e3bae27c1c8486599f3ce83"
Nov 02 22:15:03.549595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.549474    4528 generic.go:356] "Generic (PLEG): container finished" podID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerID="64fee73eff5d241fb08c42fb67292867c4fd1668ce1c20ca9b823447ffa0713f" exitCode=60
Nov 02 22:15:03.549827 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.549542    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" event={"ID":"be619aa4-7336-40a9-b5d4-1a5ba5d69129","Type":"ContainerDied","Data":"64fee73eff5d241fb08c42fb67292867c4fd1668ce1c20ca9b823447ffa0713f"}
Nov 02 22:15:03.549827 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.549596    4528 scope.go:122] "RemoveContainer" containerID="d35d17579375b1057f27b9b963e5c4efc6a23e6529a98875e488e210c3a7884d"
Nov 02 22:15:03.550031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.550014    4528 scope.go:122] "RemoveContainer" containerID="64fee73eff5d241fb08c42fb67292867c4fd1668ce1c20ca9b823447ffa0713f"
Nov 02 22:15:03.560684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.560171    4528 generic.go:356] "Generic (PLEG): container finished" podID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerID="99b158de20ab07936e58acd5a2bcbf23fb95566febe2be0719e7e6671f236eb0" exitCode=60
Nov 02 22:15:03.560684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.560279    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" event={"ID":"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1","Type":"ContainerDied","Data":"99b158de20ab07936e58acd5a2bcbf23fb95566febe2be0719e7e6671f236eb0"}
Nov 02 22:15:03.560684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.560512    4528 scope.go:122] "RemoveContainer" containerID="99b158de20ab07936e58acd5a2bcbf23fb95566febe2be0719e7e6671f236eb0"
Nov 02 22:15:03.564703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.564684    4528 generic.go:356] "Generic (PLEG): container finished" podID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerID="579f78bcdf5c5446ecfd08243a42a5aee2ae99bb876033c30c9e59c039adc0ce" exitCode=60
Nov 02 22:15:03.564779 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.564709    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" event={"ID":"0dbc0a56-99f7-4467-8501-8a50b683cb6e","Type":"ContainerDied","Data":"579f78bcdf5c5446ecfd08243a42a5aee2ae99bb876033c30c9e59c039adc0ce"}
Nov 02 22:15:03.564986 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.564973    4528 scope.go:122] "RemoveContainer" containerID="579f78bcdf5c5446ecfd08243a42a5aee2ae99bb876033c30c9e59c039adc0ce"
Nov 02 22:15:03.647080 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.646232    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" podStartSLOduration=3.646217782 podStartE2EDuration="3.646217782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.641949797 +0000 UTC m=+3990.905486115" watchObservedRunningTime="2026-11-02 22:15:03.646217782 +0000 UTC m=+3990.909754137"
Nov 02 22:15:03.754976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:03.754925    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" podStartSLOduration=3.754912089 podStartE2EDuration="3.754912089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.753805288 +0000 UTC m=+3991.017341605" watchObservedRunningTime="2026-11-02 22:15:03.754912089 +0000 UTC m=+3991.018448406"
Nov 02 22:15:04.424242 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.416849    4528 patch_prober.go:29] interesting pod/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 22:15:04.424242 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.416899    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 22:15:04.574872 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.574830    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" event={"ID":"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1","Type":"ContainerStarted","Data":"d91a18dac7f5ede981a7bd89a078400a199b3f90a3e071a4ca3b965e0f971d8c"}
Nov 02 22:15:04.577870 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.577832    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" event={"ID":"0dbc0a56-99f7-4467-8501-8a50b683cb6e","Type":"ContainerStarted","Data":"c254d8c9ebcecc2f066ef4c3a888b876f889f45dcfca023390820fa86a82785d"}
Nov 02 22:15:04.580548 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.580521    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" event={"ID":"507f7d40-be0c-4471-9821-04f1fb7480e1","Type":"ContainerStarted","Data":"59400de0a4f5c0a92f992875d53a72ce1a6dec27b9c03eeab3a7960139f09fcd"}
Nov 02 22:15:04.583170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.583142    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" event={"ID":"e5a4167b-2636-40ef-842d-9ea3cc8c46ea","Type":"ContainerStarted","Data":"9cc1ad2ad1dc3d15c4eb4de99f4b7a20131775954d0c373d1ee799c7961590d1"}
Nov 02 22:15:04.585259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.585229    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" event={"ID":"9330703a-25dd-4c73-a5b8-296354013803","Type":"ContainerStarted","Data":"fd7cba09451c985930ad7bde1a7b468b1ac4c573b4184efbcee61575775bf00e"}
Nov 02 22:15:04.587324 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.587295    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" event={"ID":"3e6cda15-3a4d-4450-b922-aa263f332786","Type":"ContainerStarted","Data":"21425733e39c359861e8596de43d2bf44bd8800afb6f8ef619b71de5e7b9b40a"}
Nov 02 22:15:04.588836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.588811    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" event={"ID":"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f","Type":"ContainerStarted","Data":"d57c6b9c6d86e0e10a43fcc1d7355c15b6ab9b36160e40c2d4d501375d9bfc6a"}
Nov 02 22:15:04.590951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.590928    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" event={"ID":"204147df-3fd6-4c5c-b2b4-691e5c45d9bc","Type":"ContainerStarted","Data":"11ed21ad602ca57af31f498cf6587d6cf66c61bc41878d811d4f7471cafb696c"}
Nov 02 22:15:04.593217 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.593196    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" event={"ID":"a21ffde3-3d62-4a69-abfc-3fecce92eb86","Type":"ContainerStarted","Data":"8de3e6ea46b04577df1d9f87164cdb67dc35322c5c0170fc5651c244c8787171"}
Nov 02 22:15:04.595367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.595345    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" event={"ID":"660176c3-ce07-49f1-af67-55c1d191a517","Type":"ContainerStarted","Data":"4b54af831bb036c530e6b510703b8d01f3a0cd765abb59074432bbad275ab680"}
Nov 02 22:15:04.597367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.597334    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" event={"ID":"b736b5d4-392b-4ee7-8757-dfe9b0b63c16","Type":"ContainerStarted","Data":"ec6c0e8cf66863e884b5f80e40c4039c7b552f4ec9d51ea73d061f98c824357a"}
Nov 02 22:15:04.599274 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.599253    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" event={"ID":"be619aa4-7336-40a9-b5d4-1a5ba5d69129","Type":"ContainerStarted","Data":"e89040c8f3a920702a46fa46080e8befab13bba7e3048c20ac6a7b2120305bc2"}
Nov 02 22:15:04.949103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:04.949072    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a" path="/var/lib/kubelet/pods/1d1b16e2-ccb4-4290-a0c1-e60810f6ac4a/volumes"
Nov 02 22:15:05.427766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.427577    4528 patch_prober.go:29] interesting pod/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 22:15:05.427766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.427635    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 22:15:05.632912 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.632880    4528 generic.go:356] "Generic (PLEG): container finished" podID="9330703a-25dd-4c73-a5b8-296354013803" containerID="fd7cba09451c985930ad7bde1a7b468b1ac4c573b4184efbcee61575775bf00e" exitCode=60
Nov 02 22:15:05.633332 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.632970    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" event={"ID":"9330703a-25dd-4c73-a5b8-296354013803","Type":"ContainerDied","Data":"fd7cba09451c985930ad7bde1a7b468b1ac4c573b4184efbcee61575775bf00e"}
Nov 02 22:15:05.633332 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.633017    4528 scope.go:122] "RemoveContainer" containerID="584d1bbd7b9c49277df72b9108fddf1f771918de72f6c058794616c238fb24ef"
Nov 02 22:15:05.633892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.633875    4528 scope.go:122] "RemoveContainer" containerID="fd7cba09451c985930ad7bde1a7b468b1ac4c573b4184efbcee61575775bf00e"
Nov 02 22:15:05.635352 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.634367    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22_odh-ai-gateway-infra(9330703a-25dd-4c73-a5b8-296354013803)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" podUID="9330703a-25dd-4c73-a5b8-296354013803"
Nov 02 22:15:05.636396 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.636369    4528 generic.go:356] "Generic (PLEG): container finished" podID="3e6cda15-3a4d-4450-b922-aa263f332786" containerID="21425733e39c359861e8596de43d2bf44bd8800afb6f8ef619b71de5e7b9b40a" exitCode=60
Nov 02 22:15:05.636476 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.636450    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" event={"ID":"3e6cda15-3a4d-4450-b922-aa263f332786","Type":"ContainerDied","Data":"21425733e39c359861e8596de43d2bf44bd8800afb6f8ef619b71de5e7b9b40a"}
Nov 02 22:15:05.636683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.636668    4528 scope.go:122] "RemoveContainer" containerID="21425733e39c359861e8596de43d2bf44bd8800afb6f8ef619b71de5e7b9b40a"
Nov 02 22:15:05.637005 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.636988    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n_odh-ai-gateway-infra(3e6cda15-3a4d-4450-b922-aa263f332786)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" podUID="3e6cda15-3a4d-4450-b922-aa263f332786"
Nov 02 22:15:05.638644 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.638624    4528 generic.go:356] "Generic (PLEG): container finished" podID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerID="d57c6b9c6d86e0e10a43fcc1d7355c15b6ab9b36160e40c2d4d501375d9bfc6a" exitCode=60
Nov 02 22:15:05.638744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.638724    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" event={"ID":"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f","Type":"ContainerDied","Data":"d57c6b9c6d86e0e10a43fcc1d7355c15b6ab9b36160e40c2d4d501375d9bfc6a"}
Nov 02 22:15:05.639353 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.639308    4528 scope.go:122] "RemoveContainer" containerID="d57c6b9c6d86e0e10a43fcc1d7355c15b6ab9b36160e40c2d4d501375d9bfc6a"
Nov 02 22:15:05.639597 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.639579    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p_odh-ai-gateway-infra(9ba9bd48-8697-47b6-9c1a-37c67b8cef4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f"
Nov 02 22:15:05.640540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.640523    4528 generic.go:356] "Generic (PLEG): container finished" podID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerID="11ed21ad602ca57af31f498cf6587d6cf66c61bc41878d811d4f7471cafb696c" exitCode=60
Nov 02 22:15:05.640619 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.640598    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" event={"ID":"204147df-3fd6-4c5c-b2b4-691e5c45d9bc","Type":"ContainerDied","Data":"11ed21ad602ca57af31f498cf6587d6cf66c61bc41878d811d4f7471cafb696c"}
Nov 02 22:15:05.640799 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.640787    4528 scope.go:122] "RemoveContainer" containerID="11ed21ad602ca57af31f498cf6587d6cf66c61bc41878d811d4f7471cafb696c"
Nov 02 22:15:05.641052 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.641033    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh_odh-ai-gateway-infra(204147df-3fd6-4c5c-b2b4-691e5c45d9bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc"
Nov 02 22:15:05.642535 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.642514    4528 generic.go:356] "Generic (PLEG): container finished" podID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerID="8de3e6ea46b04577df1d9f87164cdb67dc35322c5c0170fc5651c244c8787171" exitCode=60
Nov 02 22:15:05.642605 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.642588    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" event={"ID":"a21ffde3-3d62-4a69-abfc-3fecce92eb86","Type":"ContainerDied","Data":"8de3e6ea46b04577df1d9f87164cdb67dc35322c5c0170fc5651c244c8787171"}
Nov 02 22:15:05.642743 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.642729    4528 scope.go:122] "RemoveContainer" containerID="8de3e6ea46b04577df1d9f87164cdb67dc35322c5c0170fc5651c244c8787171"
Nov 02 22:15:05.642981 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.642964    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6_odh-ai-gateway-infra(a21ffde3-3d62-4a69-abfc-3fecce92eb86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86"
Nov 02 22:15:05.645426 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.645408    4528 generic.go:356] "Generic (PLEG): container finished" podID="660176c3-ce07-49f1-af67-55c1d191a517" containerID="4b54af831bb036c530e6b510703b8d01f3a0cd765abb59074432bbad275ab680" exitCode=60
Nov 02 22:15:05.645501 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.645489    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" event={"ID":"660176c3-ce07-49f1-af67-55c1d191a517","Type":"ContainerDied","Data":"4b54af831bb036c530e6b510703b8d01f3a0cd765abb59074432bbad275ab680"}
Nov 02 22:15:05.645685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.645666    4528 scope.go:122] "RemoveContainer" containerID="4b54af831bb036c530e6b510703b8d01f3a0cd765abb59074432bbad275ab680"
Nov 02 22:15:05.645940 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.645921    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv_odh-ai-gateway-infra(660176c3-ce07-49f1-af67-55c1d191a517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" podUID="660176c3-ce07-49f1-af67-55c1d191a517"
Nov 02 22:15:05.648365 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.648343    4528 generic.go:356] "Generic (PLEG): container finished" podID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerID="ec6c0e8cf66863e884b5f80e40c4039c7b552f4ec9d51ea73d061f98c824357a" exitCode=60
Nov 02 22:15:05.648511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.648496    4528 scope.go:122] "RemoveContainer" containerID="5186e913b1c9a32bbc996ad5f22ef23ca0fc0a2217c7a5171e22ca9a3350aa7d"
Nov 02 22:15:05.648580 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.648557    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" event={"ID":"b736b5d4-392b-4ee7-8757-dfe9b0b63c16","Type":"ContainerDied","Data":"ec6c0e8cf66863e884b5f80e40c4039c7b552f4ec9d51ea73d061f98c824357a"}
Nov 02 22:15:05.648719 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.648704    4528 scope.go:122] "RemoveContainer" containerID="ec6c0e8cf66863e884b5f80e40c4039c7b552f4ec9d51ea73d061f98c824357a"
Nov 02 22:15:05.648915 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.648897    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl_odh-ai-gateway-infra(b736b5d4-392b-4ee7-8757-dfe9b0b63c16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16"
Nov 02 22:15:05.650655 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.650638    4528 generic.go:356] "Generic (PLEG): container finished" podID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerID="e89040c8f3a920702a46fa46080e8befab13bba7e3048c20ac6a7b2120305bc2" exitCode=60
Nov 02 22:15:05.650751 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.650712    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" event={"ID":"be619aa4-7336-40a9-b5d4-1a5ba5d69129","Type":"ContainerDied","Data":"e89040c8f3a920702a46fa46080e8befab13bba7e3048c20ac6a7b2120305bc2"}
Nov 02 22:15:05.651088 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.651071    4528 scope.go:122] "RemoveContainer" containerID="e89040c8f3a920702a46fa46080e8befab13bba7e3048c20ac6a7b2120305bc2"
Nov 02 22:15:05.651439 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.651422    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4_odh-ai-gateway-infra(be619aa4-7336-40a9-b5d4-1a5ba5d69129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129"
Nov 02 22:15:05.652389 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.652372    4528 generic.go:356] "Generic (PLEG): container finished" podID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerID="d91a18dac7f5ede981a7bd89a078400a199b3f90a3e071a4ca3b965e0f971d8c" exitCode=60
Nov 02 22:15:05.652477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.652453    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" event={"ID":"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1","Type":"ContainerDied","Data":"d91a18dac7f5ede981a7bd89a078400a199b3f90a3e071a4ca3b965e0f971d8c"}
Nov 02 22:15:05.652595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.652580    4528 scope.go:122] "RemoveContainer" containerID="d91a18dac7f5ede981a7bd89a078400a199b3f90a3e071a4ca3b965e0f971d8c"
Nov 02 22:15:05.652780 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.652761    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g_odh-ai-gateway-infra(3b34b9e5-ff28-4e9f-82bd-1e002b2366a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1"
Nov 02 22:15:05.654289 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.654270    4528 generic.go:356] "Generic (PLEG): container finished" podID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerID="c254d8c9ebcecc2f066ef4c3a888b876f889f45dcfca023390820fa86a82785d" exitCode=60
Nov 02 22:15:05.654376 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.654344    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" event={"ID":"0dbc0a56-99f7-4467-8501-8a50b683cb6e","Type":"ContainerDied","Data":"c254d8c9ebcecc2f066ef4c3a888b876f889f45dcfca023390820fa86a82785d"}
Nov 02 22:15:05.654551 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.654536    4528 scope.go:122] "RemoveContainer" containerID="c254d8c9ebcecc2f066ef4c3a888b876f889f45dcfca023390820fa86a82785d"
Nov 02 22:15:05.654747 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.654726    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4_odh-ai-gateway-infra(0dbc0a56-99f7-4467-8501-8a50b683cb6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e"
Nov 02 22:15:05.656100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.656080    4528 generic.go:356] "Generic (PLEG): container finished" podID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerID="59400de0a4f5c0a92f992875d53a72ce1a6dec27b9c03eeab3a7960139f09fcd" exitCode=60
Nov 02 22:15:05.656227 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.656209    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" event={"ID":"507f7d40-be0c-4471-9821-04f1fb7480e1","Type":"ContainerDied","Data":"59400de0a4f5c0a92f992875d53a72ce1a6dec27b9c03eeab3a7960139f09fcd"}
Nov 02 22:15:05.656330 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.656312    4528 scope.go:122] "RemoveContainer" containerID="59400de0a4f5c0a92f992875d53a72ce1a6dec27b9c03eeab3a7960139f09fcd"
Nov 02 22:15:05.656558 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.656535    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-7mprd_odh-ai-gateway-infra(507f7d40-be0c-4471-9821-04f1fb7480e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1"
Nov 02 22:15:05.658333 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.658313    4528 generic.go:356] "Generic (PLEG): container finished" podID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerID="9cc1ad2ad1dc3d15c4eb4de99f4b7a20131775954d0c373d1ee799c7961590d1" exitCode=60
Nov 02 22:15:05.658458 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.658395    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" event={"ID":"e5a4167b-2636-40ef-842d-9ea3cc8c46ea","Type":"ContainerDied","Data":"9cc1ad2ad1dc3d15c4eb4de99f4b7a20131775954d0c373d1ee799c7961590d1"}
Nov 02 22:15:05.658589 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.658571    4528 scope.go:122] "RemoveContainer" containerID="9cc1ad2ad1dc3d15c4eb4de99f4b7a20131775954d0c373d1ee799c7961590d1"
Nov 02 22:15:05.658807 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:05.658788    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g_odh-ai-gateway-infra(e5a4167b-2636-40ef-842d-9ea3cc8c46ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea"
Nov 02 22:15:05.663041 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.663027    4528 scope.go:122] "RemoveContainer" containerID="25e4755adf89cd0166cf957c9c32a5b4ac2a8253d495b74043d67e4ab584cf51"
Nov 02 22:15:05.698178 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.684986    4528 scope.go:122] "RemoveContainer" containerID="9bbb8ac906e7e14d81a1831aa4ac6aa329de67369c5111d6a89b9695fcb98800"
Nov 02 22:15:05.764975 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.764423    4528 scope.go:122] "RemoveContainer" containerID="aa2ff56e8abdbe86d070f0ccd4b367a3f0eeb40183c2cef94806674d4836bb07"
Nov 02 22:15:05.780860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.780837    4528 scope.go:122] "RemoveContainer" containerID="d35d17579375b1057f27b9b963e5c4efc6a23e6529a98875e488e210c3a7884d"
Nov 02 22:15:05.797223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.797199    4528 scope.go:122] "RemoveContainer" containerID="a2f5454e1aef41b5d9048846e3c563563a592e222e3bae27c1c8486599f3ce83"
Nov 02 22:15:05.810693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.810671    4528 scope.go:122] "RemoveContainer" containerID="64fee73eff5d241fb08c42fb67292867c4fd1668ce1c20ca9b823447ffa0713f"
Nov 02 22:15:05.827032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.826992    4528 scope.go:122] "RemoveContainer" containerID="99b158de20ab07936e58acd5a2bcbf23fb95566febe2be0719e7e6671f236eb0"
Nov 02 22:15:05.841083 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.841063    4528 scope.go:122] "RemoveContainer" containerID="579f78bcdf5c5446ecfd08243a42a5aee2ae99bb876033c30c9e59c039adc0ce"
Nov 02 22:15:05.854842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.854824    4528 scope.go:122] "RemoveContainer" containerID="bea48d7653bee747f88517aaa112f416f0bdca2037401647c58878202a7a20e0"
Nov 02 22:15:05.868647 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:05.868625    4528 scope.go:122] "RemoveContainer" containerID="1e2c0202b7be13ad1e1838c9b5e9f706194d78d3496b6e4636a8572337e4fc2f"
Nov 02 22:15:06.162746 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.162692    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" podStartSLOduration=6.162674553 podStartE2EDuration="6.162674553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:06.160341007 +0000 UTC m=+3993.423877318" watchObservedRunningTime="2026-11-02 22:15:06.162674553 +0000 UTC m=+3993.426210871"
Nov 02 22:15:06.416893 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.416820    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:15:06.417911 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.417890    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:15:06.668351 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.668269    4528 scope.go:122] "RemoveContainer" containerID="59400de0a4f5c0a92f992875d53a72ce1a6dec27b9c03eeab3a7960139f09fcd"
Nov 02 22:15:06.668751 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.668608    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-7mprd_odh-ai-gateway-infra(507f7d40-be0c-4471-9821-04f1fb7480e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1"
Nov 02 22:15:06.669931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.669911    4528 scope.go:122] "RemoveContainer" containerID="9cc1ad2ad1dc3d15c4eb4de99f4b7a20131775954d0c373d1ee799c7961590d1"
Nov 02 22:15:06.670246 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.670215    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g_odh-ai-gateway-infra(e5a4167b-2636-40ef-842d-9ea3cc8c46ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea"
Nov 02 22:15:06.671540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.671515    4528 scope.go:122] "RemoveContainer" containerID="fd7cba09451c985930ad7bde1a7b468b1ac4c573b4184efbcee61575775bf00e"
Nov 02 22:15:06.671749 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.671729    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22_odh-ai-gateway-infra(9330703a-25dd-4c73-a5b8-296354013803)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" podUID="9330703a-25dd-4c73-a5b8-296354013803"
Nov 02 22:15:06.673025 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.673005    4528 scope.go:122] "RemoveContainer" containerID="21425733e39c359861e8596de43d2bf44bd8800afb6f8ef619b71de5e7b9b40a"
Nov 02 22:15:06.673260 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.673237    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n_odh-ai-gateway-infra(3e6cda15-3a4d-4450-b922-aa263f332786)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" podUID="3e6cda15-3a4d-4450-b922-aa263f332786"
Nov 02 22:15:06.674512 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.674494    4528 scope.go:122] "RemoveContainer" containerID="d57c6b9c6d86e0e10a43fcc1d7355c15b6ab9b36160e40c2d4d501375d9bfc6a"
Nov 02 22:15:06.674704 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.674674    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p_odh-ai-gateway-infra(9ba9bd48-8697-47b6-9c1a-37c67b8cef4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f"
Nov 02 22:15:06.676055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.676039    4528 scope.go:122] "RemoveContainer" containerID="11ed21ad602ca57af31f498cf6587d6cf66c61bc41878d811d4f7471cafb696c"
Nov 02 22:15:06.676261 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.676243    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh_odh-ai-gateway-infra(204147df-3fd6-4c5c-b2b4-691e5c45d9bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc"
Nov 02 22:15:06.677561 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.677543    4528 scope.go:122] "RemoveContainer" containerID="8de3e6ea46b04577df1d9f87164cdb67dc35322c5c0170fc5651c244c8787171"
Nov 02 22:15:06.677760 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.677738    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6_odh-ai-gateway-infra(a21ffde3-3d62-4a69-abfc-3fecce92eb86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86"
Nov 02 22:15:06.679018 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.679002    4528 scope.go:122] "RemoveContainer" containerID="4b54af831bb036c530e6b510703b8d01f3a0cd765abb59074432bbad275ab680"
Nov 02 22:15:06.679242 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.679220    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv_odh-ai-gateway-infra(660176c3-ce07-49f1-af67-55c1d191a517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" podUID="660176c3-ce07-49f1-af67-55c1d191a517"
Nov 02 22:15:06.680529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.680512    4528 scope.go:122] "RemoveContainer" containerID="ec6c0e8cf66863e884b5f80e40c4039c7b552f4ec9d51ea73d061f98c824357a"
Nov 02 22:15:06.680726 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.680705    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl_odh-ai-gateway-infra(b736b5d4-392b-4ee7-8757-dfe9b0b63c16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16"
Nov 02 22:15:06.681946 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.681926    4528 scope.go:122] "RemoveContainer" containerID="e89040c8f3a920702a46fa46080e8befab13bba7e3048c20ac6a7b2120305bc2"
Nov 02 22:15:06.682163 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.682146    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4_odh-ai-gateway-infra(be619aa4-7336-40a9-b5d4-1a5ba5d69129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129"
Nov 02 22:15:06.683403 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.683385    4528 scope.go:122] "RemoveContainer" containerID="d91a18dac7f5ede981a7bd89a078400a199b3f90a3e071a4ca3b965e0f971d8c"
Nov 02 22:15:06.683582 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.683562    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g_odh-ai-gateway-infra(3b34b9e5-ff28-4e9f-82bd-1e002b2366a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1"
Nov 02 22:15:06.684822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:06.684807    4528 scope.go:122] "RemoveContainer" containerID="c254d8c9ebcecc2f066ef4c3a888b876f889f45dcfca023390820fa86a82785d"
Nov 02 22:15:06.684989 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:06.684973    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4_odh-ai-gateway-infra(0dbc0a56-99f7-4467-8501-8a50b683cb6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e"
Nov 02 22:15:14.114149 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:14.114094    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:15:15.906326 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:15.906290    4528 scope.go:122] "RemoveContainer" containerID="d57c6b9c6d86e0e10a43fcc1d7355c15b6ab9b36160e40c2d4d501375d9bfc6a"
Nov 02 22:15:15.906730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:15.906432    4528 scope.go:122] "RemoveContainer" containerID="e89040c8f3a920702a46fa46080e8befab13bba7e3048c20ac6a7b2120305bc2"
Nov 02 22:15:15.906730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:15.906626    4528 scope.go:122] "RemoveContainer" containerID="11ed21ad602ca57af31f498cf6587d6cf66c61bc41878d811d4f7471cafb696c"
Nov 02 22:15:16.774184 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:16.774149    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" event={"ID":"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f","Type":"ContainerStarted","Data":"fad2f74f923089c26251f3485f0510a7a3060a1d0b5722815b3dff1848262290"}
Nov 02 22:15:16.776159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:16.776130    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" event={"ID":"204147df-3fd6-4c5c-b2b4-691e5c45d9bc","Type":"ContainerStarted","Data":"9a0ad87f425e97dd09152e05aefc7e628ed0398de6a0ef50af8a077c56794c1d"}
Nov 02 22:15:16.778139 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:16.778097    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" event={"ID":"be619aa4-7336-40a9-b5d4-1a5ba5d69129","Type":"ContainerStarted","Data":"2bb119b3b8bc2530f7317f5bda9e8a69f01fcef72efb546612e273e14f286c92"}
Nov 02 22:15:16.908548 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:16.908515    4528 scope.go:122] "RemoveContainer" containerID="d91a18dac7f5ede981a7bd89a078400a199b3f90a3e071a4ca3b965e0f971d8c"
Nov 02 22:15:16.909610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:16.909580    4528 scope.go:122] "RemoveContainer" containerID="8de3e6ea46b04577df1d9f87164cdb67dc35322c5c0170fc5651c244c8787171"
Nov 02 22:15:16.913750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:16.913200    4528 scope.go:122] "RemoveContainer" containerID="59400de0a4f5c0a92f992875d53a72ce1a6dec27b9c03eeab3a7960139f09fcd"
Nov 02 22:15:17.787299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.787259    4528 generic.go:356] "Generic (PLEG): container finished" podID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerID="fad2f74f923089c26251f3485f0510a7a3060a1d0b5722815b3dff1848262290" exitCode=60
Nov 02 22:15:17.787514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.787329    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" event={"ID":"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f","Type":"ContainerDied","Data":"fad2f74f923089c26251f3485f0510a7a3060a1d0b5722815b3dff1848262290"}
Nov 02 22:15:17.787514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.787398    4528 scope.go:122] "RemoveContainer" containerID="d57c6b9c6d86e0e10a43fcc1d7355c15b6ab9b36160e40c2d4d501375d9bfc6a"
Nov 02 22:15:17.787752 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.787736    4528 scope.go:122] "RemoveContainer" containerID="fad2f74f923089c26251f3485f0510a7a3060a1d0b5722815b3dff1848262290"
Nov 02 22:15:17.788120 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:17.788067    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p_odh-ai-gateway-infra(9ba9bd48-8697-47b6-9c1a-37c67b8cef4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f"
Nov 02 22:15:17.789548 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.789529    4528 generic.go:356] "Generic (PLEG): container finished" podID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerID="9a0ad87f425e97dd09152e05aefc7e628ed0398de6a0ef50af8a077c56794c1d" exitCode=60
Nov 02 22:15:17.789653 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.789577    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" event={"ID":"204147df-3fd6-4c5c-b2b4-691e5c45d9bc","Type":"ContainerDied","Data":"9a0ad87f425e97dd09152e05aefc7e628ed0398de6a0ef50af8a077c56794c1d"}
Nov 02 22:15:17.789963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.789948    4528 scope.go:122] "RemoveContainer" containerID="9a0ad87f425e97dd09152e05aefc7e628ed0398de6a0ef50af8a077c56794c1d"
Nov 02 22:15:17.790306 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:17.790277    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh_odh-ai-gateway-infra(204147df-3fd6-4c5c-b2b4-691e5c45d9bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc"
Nov 02 22:15:17.791649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.791620    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" event={"ID":"a21ffde3-3d62-4a69-abfc-3fecce92eb86","Type":"ContainerStarted","Data":"04dd168c1f70102b2b7658a0950a08fd4cafe79615c5126d7b746ddd198fa2cf"}
Nov 02 22:15:17.793668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.793647    4528 generic.go:356] "Generic (PLEG): container finished" podID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerID="2bb119b3b8bc2530f7317f5bda9e8a69f01fcef72efb546612e273e14f286c92" exitCode=60
Nov 02 22:15:17.793780 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.793689    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" event={"ID":"be619aa4-7336-40a9-b5d4-1a5ba5d69129","Type":"ContainerDied","Data":"2bb119b3b8bc2530f7317f5bda9e8a69f01fcef72efb546612e273e14f286c92"}
Nov 02 22:15:17.794040 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.794020    4528 scope.go:122] "RemoveContainer" containerID="2bb119b3b8bc2530f7317f5bda9e8a69f01fcef72efb546612e273e14f286c92"
Nov 02 22:15:17.794498 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:17.794319    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4_odh-ai-gateway-infra(be619aa4-7336-40a9-b5d4-1a5ba5d69129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129"
Nov 02 22:15:17.795850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.795829    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" event={"ID":"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1","Type":"ContainerStarted","Data":"4bd5e5002e61dfd5191aeac8ab9d3db9ca0c1c1f3380d2857d303447ba609f3d"}
Nov 02 22:15:17.797926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.797899    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" event={"ID":"507f7d40-be0c-4471-9821-04f1fb7480e1","Type":"ContainerStarted","Data":"c612c6d8c5efd0c5d612daf123653753f76f0ce4db8484874f5b3d507f661808"}
Nov 02 22:15:17.801472 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.801443    4528 scope.go:122] "RemoveContainer" containerID="11ed21ad602ca57af31f498cf6587d6cf66c61bc41878d811d4f7471cafb696c"
Nov 02 22:15:17.815057 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.815037    4528 scope.go:122] "RemoveContainer" containerID="e89040c8f3a920702a46fa46080e8befab13bba7e3048c20ac6a7b2120305bc2"
Nov 02 22:15:17.900934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.900845    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"]
Nov 02 22:15:17.907092 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:17.907026    4528 scope.go:122] "RemoveContainer" containerID="4b54af831bb036c530e6b510703b8d01f3a0cd765abb59074432bbad275ab680"
Nov 02 22:15:18.013233 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.013208    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"]
Nov 02 22:15:18.126436 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.126407    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"]
Nov 02 22:15:18.814288 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.814257    4528 generic.go:356] "Generic (PLEG): container finished" podID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerID="04dd168c1f70102b2b7658a0950a08fd4cafe79615c5126d7b746ddd198fa2cf" exitCode=60
Nov 02 22:15:18.814428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.814332    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" event={"ID":"a21ffde3-3d62-4a69-abfc-3fecce92eb86","Type":"ContainerDied","Data":"04dd168c1f70102b2b7658a0950a08fd4cafe79615c5126d7b746ddd198fa2cf"}
Nov 02 22:15:18.814428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.814410    4528 scope.go:122] "RemoveContainer" containerID="8de3e6ea46b04577df1d9f87164cdb67dc35322c5c0170fc5651c244c8787171"
Nov 02 22:15:18.814885 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.814866    4528 scope.go:122] "RemoveContainer" containerID="04dd168c1f70102b2b7658a0950a08fd4cafe79615c5126d7b746ddd198fa2cf"
Nov 02 22:15:18.815322 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:18.815291    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6_odh-ai-gateway-infra(a21ffde3-3d62-4a69-abfc-3fecce92eb86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86"
Nov 02 22:15:18.816681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.816648    4528 generic.go:356] "Generic (PLEG): container finished" podID="660176c3-ce07-49f1-af67-55c1d191a517" containerID="e4ee35b0f7a37b01b6772f8cf23b9937240531c13105a9e47377df41efcc60c0" exitCode=60
Nov 02 22:15:18.816833 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.816817    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" event={"ID":"660176c3-ce07-49f1-af67-55c1d191a517","Type":"ContainerDied","Data":"e4ee35b0f7a37b01b6772f8cf23b9937240531c13105a9e47377df41efcc60c0"}
Nov 02 22:15:18.817269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.817248    4528 scope.go:122] "RemoveContainer" containerID="e4ee35b0f7a37b01b6772f8cf23b9937240531c13105a9e47377df41efcc60c0"
Nov 02 22:15:18.817628 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:18.817604    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv_odh-ai-gateway-infra(660176c3-ce07-49f1-af67-55c1d191a517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" podUID="660176c3-ce07-49f1-af67-55c1d191a517"
Nov 02 22:15:18.822663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.822638    4528 generic.go:356] "Generic (PLEG): container finished" podID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerID="4bd5e5002e61dfd5191aeac8ab9d3db9ca0c1c1f3380d2857d303447ba609f3d" exitCode=60
Nov 02 22:15:18.822805 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.822764    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" event={"ID":"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1","Type":"ContainerDied","Data":"4bd5e5002e61dfd5191aeac8ab9d3db9ca0c1c1f3380d2857d303447ba609f3d"}
Nov 02 22:15:18.823226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.823198    4528 scope.go:122] "RemoveContainer" containerID="4bd5e5002e61dfd5191aeac8ab9d3db9ca0c1c1f3380d2857d303447ba609f3d"
Nov 02 22:15:18.823564 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:18.823539    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g_odh-ai-gateway-infra(3b34b9e5-ff28-4e9f-82bd-1e002b2366a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1"
Nov 02 22:15:18.826064 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.826002    4528 generic.go:356] "Generic (PLEG): container finished" podID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerID="c612c6d8c5efd0c5d612daf123653753f76f0ce4db8484874f5b3d507f661808" exitCode=60
Nov 02 22:15:18.826200 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.826061    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" event={"ID":"507f7d40-be0c-4471-9821-04f1fb7480e1","Type":"ContainerDied","Data":"c612c6d8c5efd0c5d612daf123653753f76f0ce4db8484874f5b3d507f661808"}
Nov 02 22:15:18.826484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.826468    4528 scope.go:122] "RemoveContainer" containerID="c612c6d8c5efd0c5d612daf123653753f76f0ce4db8484874f5b3d507f661808"
Nov 02 22:15:18.826876 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:18.826782    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-7mprd_odh-ai-gateway-infra(507f7d40-be0c-4471-9821-04f1fb7480e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1"
Nov 02 22:15:18.830688 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.830593    4528 scope.go:122] "RemoveContainer" containerID="4b54af831bb036c530e6b510703b8d01f3a0cd765abb59074432bbad275ab680"
Nov 02 22:15:18.847736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.847422    4528 scope.go:122] "RemoveContainer" containerID="d91a18dac7f5ede981a7bd89a078400a199b3f90a3e071a4ca3b965e0f971d8c"
Nov 02 22:15:18.866511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.866481    4528 scope.go:122] "RemoveContainer" containerID="59400de0a4f5c0a92f992875d53a72ce1a6dec27b9c03eeab3a7960139f09fcd"
Nov 02 22:15:18.907226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.907195    4528 scope.go:122] "RemoveContainer" containerID="fd7cba09451c985930ad7bde1a7b468b1ac4c573b4184efbcee61575775bf00e"
Nov 02 22:15:18.913128 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.908082    4528 scope.go:122] "RemoveContainer" containerID="ec6c0e8cf66863e884b5f80e40c4039c7b552f4ec9d51ea73d061f98c824357a"
Nov 02 22:15:18.913128 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.908805    4528 scope.go:122] "RemoveContainer" containerID="9cc1ad2ad1dc3d15c4eb4de99f4b7a20131775954d0c373d1ee799c7961590d1"
Nov 02 22:15:18.913128 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.909456    4528 scope.go:122] "RemoveContainer" containerID="21425733e39c359861e8596de43d2bf44bd8800afb6f8ef619b71de5e7b9b40a"
Nov 02 22:15:18.913128 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.910036    4528 scope.go:122] "RemoveContainer" containerID="c254d8c9ebcecc2f066ef4c3a888b876f889f45dcfca023390820fa86a82785d"
Nov 02 22:15:18.991943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:18.990935    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:19.040799 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.037960    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"]
Nov 02 22:15:19.092096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.092062    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-maas-api-service-ca\") pod \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\" (UID: \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\") "
Nov 02 22:15:19.092297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.092265    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-kube-api-access-xwr9g\" (UniqueName: \"kubernetes.io/projected/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-kube-api-access-xwr9g\") pod \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\" (UID: \"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f\") "
Nov 02 22:15:19.092441 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.092413    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-maas-api-service-ca" pod "9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" (UID: "9ba9bd48-8697-47b6-9c1a-37c67b8cef4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:19.093400 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.093371    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:19.094939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.094911    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-kube-api-access-xwr9g" pod "9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" (UID: "9ba9bd48-8697-47b6-9c1a-37c67b8cef4f"). InnerVolumeSpecName "kube-api-access-xwr9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:19.153996 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.153961    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"]
Nov 02 22:15:19.194211 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.194179    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwr9g\" (UniqueName: \"kubernetes.io/projected/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f-kube-api-access-xwr9g\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:19.264740 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.264718    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:19.267281 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.267262    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:19.396306 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.396273    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be619aa4-7336-40a9-b5d4-1a5ba5d69129-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be619aa4-7336-40a9-b5d4-1a5ba5d69129-maas-api-service-ca\") pod \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\" (UID: \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\") "
Nov 02 22:15:19.396306 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.396309    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-kube-api-access-6zq5w\" (UniqueName: \"kubernetes.io/projected/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-kube-api-access-6zq5w\") pod \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\" (UID: \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\") "
Nov 02 22:15:19.396577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.396423    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-maas-api-service-ca\") pod \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\" (UID: \"204147df-3fd6-4c5c-b2b4-691e5c45d9bc\") "
Nov 02 22:15:19.396577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.396555    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be619aa4-7336-40a9-b5d4-1a5ba5d69129-kube-api-access-jqbvt\" (UniqueName: \"kubernetes.io/projected/be619aa4-7336-40a9-b5d4-1a5ba5d69129-kube-api-access-jqbvt\") pod \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\" (UID: \"be619aa4-7336-40a9-b5d4-1a5ba5d69129\") "
Nov 02 22:15:19.396690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.396649    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be619aa4-7336-40a9-b5d4-1a5ba5d69129-maas-api-service-ca" pod "be619aa4-7336-40a9-b5d4-1a5ba5d69129" (UID: "be619aa4-7336-40a9-b5d4-1a5ba5d69129"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:19.396853 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.396829    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-maas-api-service-ca" pod "204147df-3fd6-4c5c-b2b4-691e5c45d9bc" (UID: "204147df-3fd6-4c5c-b2b4-691e5c45d9bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:19.397148 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.397132    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be619aa4-7336-40a9-b5d4-1a5ba5d69129-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:19.397237 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.397149    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:19.402282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.402257    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be619aa4-7336-40a9-b5d4-1a5ba5d69129-kube-api-access-jqbvt" pod "be619aa4-7336-40a9-b5d4-1a5ba5d69129" (UID: "be619aa4-7336-40a9-b5d4-1a5ba5d69129"). InnerVolumeSpecName "kube-api-access-jqbvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:19.402429 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.402322    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-kube-api-access-6zq5w" pod "204147df-3fd6-4c5c-b2b4-691e5c45d9bc" (UID: "204147df-3fd6-4c5c-b2b4-691e5c45d9bc"). InnerVolumeSpecName "kube-api-access-6zq5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:19.483910 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.483876    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"]
Nov 02 22:15:19.498101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.498063    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqbvt\" (UniqueName: \"kubernetes.io/projected/be619aa4-7336-40a9-b5d4-1a5ba5d69129-kube-api-access-jqbvt\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:19.498101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.498095    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zq5w\" (UniqueName: \"kubernetes.io/projected/204147df-3fd6-4c5c-b2b4-691e5c45d9bc-kube-api-access-6zq5w\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:19.838491 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.838451    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4" event={"ID":"be619aa4-7336-40a9-b5d4-1a5ba5d69129","Type":"ContainerDied","Data":"d326a5ca97f9f08eee83f6001b2c9b0b7c395688f5770eb3393833eb4a646e4e"}
Nov 02 22:15:19.838708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.838525    4528 scope.go:122] "RemoveContainer" containerID="2bb119b3b8bc2530f7317f5bda9e8a69f01fcef72efb546612e273e14f286c92"
Nov 02 22:15:19.838708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.838638    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"
Nov 02 22:15:19.845299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.845269    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" event={"ID":"0dbc0a56-99f7-4467-8501-8a50b683cb6e","Type":"ContainerStarted","Data":"f349d95c5f46571bf7326b172bbf7a8a5518871e78ae5c4e5daf4790f7494a8a"}
Nov 02 22:15:19.856411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.856381    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" event={"ID":"e5a4167b-2636-40ef-842d-9ea3cc8c46ea","Type":"ContainerStarted","Data":"8daedfd0048c538cfc0206a680f1c886c2f22ea3ee06b91d8934648c68f59e01"}
Nov 02 22:15:19.860489 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.860461    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" event={"ID":"9330703a-25dd-4c73-a5b8-296354013803","Type":"ContainerStarted","Data":"e6cfb8010b736b86d604ea8af3e47c3e52e8cf3202ec9fdeb6b3367748442e1c"}
Nov 02 22:15:19.862929 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.862903    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" event={"ID":"3e6cda15-3a4d-4450-b922-aa263f332786","Type":"ContainerStarted","Data":"0a2138080a4cac5a601826510451d96934bcbbb1d15f64cf90a9007af83d3cc6"}
Nov 02 22:15:19.865170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.865144    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p" event={"ID":"9ba9bd48-8697-47b6-9c1a-37c67b8cef4f","Type":"ContainerDied","Data":"4ad9937b6d2191af65711c9ad6cfe219083dd652084a9cc5b47769860c8bb6ce"}
Nov 02 22:15:19.865308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.865187    4528 scope.go:122] "RemoveContainer" containerID="fad2f74f923089c26251f3485f0510a7a3060a1d0b5722815b3dff1848262290"
Nov 02 22:15:19.865308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:19.865275    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"
Nov 02 22:15:20.051437 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.049346    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh" event={"ID":"204147df-3fd6-4c5c-b2b4-691e5c45d9bc","Type":"ContainerDied","Data":"bb03d93ff566f85addd2d6bae6ff5d3faa6a1fadb4194a9c192cdc25ec96d05c"}
Nov 02 22:15:20.051437 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.049474    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"
Nov 02 22:15:20.060886 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.060792    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"]
Nov 02 22:15:20.089859 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.088818    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" event={"ID":"b736b5d4-392b-4ee7-8757-dfe9b0b63c16","Type":"ContainerStarted","Data":"e66f58bf0dd3065599eab6a163df233e2f3b79aa5743d04636c12f092fccb412"}
Nov 02 22:15:20.276564 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.261285    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:15:20.403165 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.401934    4528 scope.go:122] "RemoveContainer" containerID="9a0ad87f425e97dd09152e05aefc7e628ed0398de6a0ef50af8a077c56794c1d"
Nov 02 22:15:20.537382 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.537131    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:20.539484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.539399    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:20.601707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.597218    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507f7d40-be0c-4471-9821-04f1fb7480e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f7d40-be0c-4471-9821-04f1fb7480e1-maas-api-service-ca\") pod \"507f7d40-be0c-4471-9821-04f1fb7480e1\" (UID: \"507f7d40-be0c-4471-9821-04f1fb7480e1\") "
Nov 02 22:15:20.601707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.597364    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507f7d40-be0c-4471-9821-04f1fb7480e1-kube-api-access-9qb64\" (UniqueName: \"kubernetes.io/projected/507f7d40-be0c-4471-9821-04f1fb7480e1-kube-api-access-9qb64\") pod \"507f7d40-be0c-4471-9821-04f1fb7480e1\" (UID: \"507f7d40-be0c-4471-9821-04f1fb7480e1\") "
Nov 02 22:15:20.601707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.597436    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-maas-api-service-ca\") pod \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\" (UID: \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\") "
Nov 02 22:15:20.601707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.597456    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-kube-api-access-tk286\" (UniqueName: \"kubernetes.io/projected/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-kube-api-access-tk286\") pod \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\" (UID: \"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1\") "
Nov 02 22:15:20.601707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.599685    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507f7d40-be0c-4471-9821-04f1fb7480e1-maas-api-service-ca" pod "507f7d40-be0c-4471-9821-04f1fb7480e1" (UID: "507f7d40-be0c-4471-9821-04f1fb7480e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:20.601707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.599987    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-maas-api-service-ca" pod "3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" (UID: "3b34b9e5-ff28-4e9f-82bd-1e002b2366a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:20.601707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.601636    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-kube-api-access-tk286" pod "3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" (UID: "3b34b9e5-ff28-4e9f-82bd-1e002b2366a1"). InnerVolumeSpecName "kube-api-access-tk286". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:20.605599 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.605044    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507f7d40-be0c-4471-9821-04f1fb7480e1-kube-api-access-9qb64" pod "507f7d40-be0c-4471-9821-04f1fb7480e1" (UID: "507f7d40-be0c-4471-9821-04f1fb7480e1"). InnerVolumeSpecName "kube-api-access-9qb64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:20.617484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.617315    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:20.682607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.682579    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:20.698675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.698636    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21ffde3-3d62-4a69-abfc-3fecce92eb86-kube-api-access-45rj5\" (UniqueName: \"kubernetes.io/projected/a21ffde3-3d62-4a69-abfc-3fecce92eb86-kube-api-access-45rj5\") pod \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\" (UID: \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\") "
Nov 02 22:15:20.698886 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.698757    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21ffde3-3d62-4a69-abfc-3fecce92eb86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21ffde3-3d62-4a69-abfc-3fecce92eb86-maas-api-service-ca\") pod \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\" (UID: \"a21ffde3-3d62-4a69-abfc-3fecce92eb86\") "
Nov 02 22:15:20.698886 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.698831    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660176c3-ce07-49f1-af67-55c1d191a517-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660176c3-ce07-49f1-af67-55c1d191a517-maas-api-service-ca\") pod \"660176c3-ce07-49f1-af67-55c1d191a517\" (UID: \"660176c3-ce07-49f1-af67-55c1d191a517\") "
Nov 02 22:15:20.699022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.698988    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660176c3-ce07-49f1-af67-55c1d191a517-kube-api-access-dvn4k\" (UniqueName: \"kubernetes.io/projected/660176c3-ce07-49f1-af67-55c1d191a517-kube-api-access-dvn4k\") pod \"660176c3-ce07-49f1-af67-55c1d191a517\" (UID: \"660176c3-ce07-49f1-af67-55c1d191a517\") "
Nov 02 22:15:20.699362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.699220    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21ffde3-3d62-4a69-abfc-3fecce92eb86-maas-api-service-ca" pod "a21ffde3-3d62-4a69-abfc-3fecce92eb86" (UID: "a21ffde3-3d62-4a69-abfc-3fecce92eb86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:20.699577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.699467    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660176c3-ce07-49f1-af67-55c1d191a517-maas-api-service-ca" pod "660176c3-ce07-49f1-af67-55c1d191a517" (UID: "660176c3-ce07-49f1-af67-55c1d191a517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:20.699577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.699540    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21ffde3-3d62-4a69-abfc-3fecce92eb86-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:20.699577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.699560    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qb64\" (UniqueName: \"kubernetes.io/projected/507f7d40-be0c-4471-9821-04f1fb7480e1-kube-api-access-9qb64\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:20.699577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.699572    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:20.699818 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.699585    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk286\" (UniqueName: \"kubernetes.io/projected/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1-kube-api-access-tk286\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:20.699818 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.699599    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f7d40-be0c-4471-9821-04f1fb7480e1-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:20.701617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.701581    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21ffde3-3d62-4a69-abfc-3fecce92eb86-kube-api-access-45rj5" pod "a21ffde3-3d62-4a69-abfc-3fecce92eb86" (UID: "a21ffde3-3d62-4a69-abfc-3fecce92eb86"). InnerVolumeSpecName "kube-api-access-45rj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:20.702690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.702645    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660176c3-ce07-49f1-af67-55c1d191a517-kube-api-access-dvn4k" pod "660176c3-ce07-49f1-af67-55c1d191a517" (UID: "660176c3-ce07-49f1-af67-55c1d191a517"). InnerVolumeSpecName "kube-api-access-dvn4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:20.808391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.808348    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660176c3-ce07-49f1-af67-55c1d191a517-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:20.830166 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.808660    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvn4k\" (UniqueName: \"kubernetes.io/projected/660176c3-ce07-49f1-af67-55c1d191a517-kube-api-access-dvn4k\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:20.830485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:20.830452    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45rj5\" (UniqueName: \"kubernetes.io/projected/a21ffde3-3d62-4a69-abfc-3fecce92eb86-kube-api-access-45rj5\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:21.055141 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.053453    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"]
Nov 02 22:15:21.092944 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.091922    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894295-l5lj4"]
Nov 02 22:15:21.115158 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.115128    4528 generic.go:356] "Generic (PLEG): container finished" podID="9330703a-25dd-4c73-a5b8-296354013803" containerID="e6cfb8010b736b86d604ea8af3e47c3e52e8cf3202ec9fdeb6b3367748442e1c" exitCode=60
Nov 02 22:15:21.115316 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.115193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" event={"ID":"9330703a-25dd-4c73-a5b8-296354013803","Type":"ContainerDied","Data":"e6cfb8010b736b86d604ea8af3e47c3e52e8cf3202ec9fdeb6b3367748442e1c"}
Nov 02 22:15:21.115316 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.115276    4528 scope.go:122] "RemoveContainer" containerID="fd7cba09451c985930ad7bde1a7b468b1ac4c573b4184efbcee61575775bf00e"
Nov 02 22:15:21.115688 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.115672    4528 scope.go:122] "RemoveContainer" containerID="e6cfb8010b736b86d604ea8af3e47c3e52e8cf3202ec9fdeb6b3367748442e1c"
Nov 02 22:15:21.115977 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:21.115960    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22_odh-ai-gateway-infra(9330703a-25dd-4c73-a5b8-296354013803)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" podUID="9330703a-25dd-4c73-a5b8-296354013803"
Nov 02 22:15:21.117075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.117056    4528 generic.go:356] "Generic (PLEG): container finished" podID="3e6cda15-3a4d-4450-b922-aa263f332786" containerID="0a2138080a4cac5a601826510451d96934bcbbb1d15f64cf90a9007af83d3cc6" exitCode=60
Nov 02 22:15:21.117164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.117143    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" event={"ID":"3e6cda15-3a4d-4450-b922-aa263f332786","Type":"ContainerDied","Data":"0a2138080a4cac5a601826510451d96934bcbbb1d15f64cf90a9007af83d3cc6"}
Nov 02 22:15:21.118432 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.118415    4528 scope.go:122] "RemoveContainer" containerID="0a2138080a4cac5a601826510451d96934bcbbb1d15f64cf90a9007af83d3cc6"
Nov 02 22:15:21.118798 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:21.118767    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n_odh-ai-gateway-infra(3e6cda15-3a4d-4450-b922-aa263f332786)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" podUID="3e6cda15-3a4d-4450-b922-aa263f332786"
Nov 02 22:15:21.121376 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.121355    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6" event={"ID":"a21ffde3-3d62-4a69-abfc-3fecce92eb86","Type":"ContainerDied","Data":"7142c8fc1cab5823bf208e9565839f0b9bbb041443e035fcbd73c7f946c00f8c"}
Nov 02 22:15:21.122566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.122550    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"
Nov 02 22:15:21.122927 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.122909    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv" event={"ID":"660176c3-ce07-49f1-af67-55c1d191a517","Type":"ContainerDied","Data":"77f06014bfd653bb19736e1d044b0eba2362ef0c69ba938d2e8f309b1579da2b"}
Nov 02 22:15:21.123362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.123342    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"
Nov 02 22:15:21.124858 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.124832    4528 generic.go:356] "Generic (PLEG): container finished" podID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerID="e66f58bf0dd3065599eab6a163df233e2f3b79aa5743d04636c12f092fccb412" exitCode=60
Nov 02 22:15:21.124986 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.124912    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" event={"ID":"b736b5d4-392b-4ee7-8757-dfe9b0b63c16","Type":"ContainerDied","Data":"e66f58bf0dd3065599eab6a163df233e2f3b79aa5743d04636c12f092fccb412"}
Nov 02 22:15:21.125524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.125218    4528 scope.go:122] "RemoveContainer" containerID="e66f58bf0dd3065599eab6a163df233e2f3b79aa5743d04636c12f092fccb412"
Nov 02 22:15:21.125524 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:21.125402    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl_odh-ai-gateway-infra(b736b5d4-392b-4ee7-8757-dfe9b0b63c16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16"
Nov 02 22:15:21.127863 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.127846    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"
Nov 02 22:15:21.128049 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.127860    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g" event={"ID":"3b34b9e5-ff28-4e9f-82bd-1e002b2366a1","Type":"ContainerDied","Data":"73f6189ce23cfaad8693e77b3747bb6ef26d7842076c232fe348203a85c4aff2"}
Nov 02 22:15:21.130196 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.130171    4528 generic.go:356] "Generic (PLEG): container finished" podID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerID="f349d95c5f46571bf7326b172bbf7a8a5518871e78ae5c4e5daf4790f7494a8a" exitCode=60
Nov 02 22:15:21.130305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.130253    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" event={"ID":"0dbc0a56-99f7-4467-8501-8a50b683cb6e","Type":"ContainerDied","Data":"f349d95c5f46571bf7326b172bbf7a8a5518871e78ae5c4e5daf4790f7494a8a"}
Nov 02 22:15:21.130750 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.130651    4528 scope.go:122] "RemoveContainer" containerID="f349d95c5f46571bf7326b172bbf7a8a5518871e78ae5c4e5daf4790f7494a8a"
Nov 02 22:15:21.130958 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:21.130907    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4_odh-ai-gateway-infra(0dbc0a56-99f7-4467-8501-8a50b683cb6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e"
Nov 02 22:15:21.132452 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.132432    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"
Nov 02 22:15:21.132955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.132928    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd" event={"ID":"507f7d40-be0c-4471-9821-04f1fb7480e1","Type":"ContainerDied","Data":"04566f4fc6281eedc0d556afec1686826c9f66f5d4dd6e6b9ad325313a692a2e"}
Nov 02 22:15:21.133819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.133802    4528 scope.go:122] "RemoveContainer" containerID="21425733e39c359861e8596de43d2bf44bd8800afb6f8ef619b71de5e7b9b40a"
Nov 02 22:15:21.135943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.135922    4528 generic.go:356] "Generic (PLEG): container finished" podID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerID="8daedfd0048c538cfc0206a680f1c886c2f22ea3ee06b91d8934648c68f59e01" exitCode=60
Nov 02 22:15:21.136097 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.135993    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" event={"ID":"e5a4167b-2636-40ef-842d-9ea3cc8c46ea","Type":"ContainerDied","Data":"8daedfd0048c538cfc0206a680f1c886c2f22ea3ee06b91d8934648c68f59e01"}
Nov 02 22:15:21.136494 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.136476    4528 scope.go:122] "RemoveContainer" containerID="8daedfd0048c538cfc0206a680f1c886c2f22ea3ee06b91d8934648c68f59e01"
Nov 02 22:15:21.136708 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:21.136690    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g_odh-ai-gateway-infra(e5a4167b-2636-40ef-842d-9ea3cc8c46ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea"
Nov 02 22:15:21.149959 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.149928    4528 scope.go:122] "RemoveContainer" containerID="04dd168c1f70102b2b7658a0950a08fd4cafe79615c5126d7b746ddd198fa2cf"
Nov 02 22:15:21.162229 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.162207    4528 scope.go:122] "RemoveContainer" containerID="e4ee35b0f7a37b01b6772f8cf23b9937240531c13105a9e47377df41efcc60c0"
Nov 02 22:15:21.172052 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.172020    4528 scope.go:122] "RemoveContainer" containerID="ec6c0e8cf66863e884b5f80e40c4039c7b552f4ec9d51ea73d061f98c824357a"
Nov 02 22:15:21.187847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.187826    4528 scope.go:122] "RemoveContainer" containerID="4bd5e5002e61dfd5191aeac8ab9d3db9ca0c1c1f3380d2857d303447ba609f3d"
Nov 02 22:15:21.200268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.200246    4528 scope.go:122] "RemoveContainer" containerID="c254d8c9ebcecc2f066ef4c3a888b876f889f45dcfca023390820fa86a82785d"
Nov 02 22:15:21.214347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.214326    4528 scope.go:122] "RemoveContainer" containerID="c612c6d8c5efd0c5d612daf123653753f76f0ce4db8484874f5b3d507f661808"
Nov 02 22:15:21.223429 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.223407    4528 scope.go:122] "RemoveContainer" containerID="9cc1ad2ad1dc3d15c4eb4de99f4b7a20131775954d0c373d1ee799c7961590d1"
Nov 02 22:15:21.430664 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.430621    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"]
Nov 02 22:15:21.554094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.554060    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"]
Nov 02 22:15:21.589730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.589694    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"]
Nov 02 22:15:21.593154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:21.593126    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-065be7-29894295-cmg5p"]
Nov 02 22:15:22.147241 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.145512    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"]
Nov 02 22:15:22.157809 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.157787    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"]
Nov 02 22:15:22.161597 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.161575    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894295-blpdv"]
Nov 02 22:15:22.258955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.255833    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"]
Nov 02 22:15:22.292257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.292218    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"]
Nov 02 22:15:22.297924 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.297892    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894295-7txrh"]
Nov 02 22:15:22.369625 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.369590    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"]
Nov 02 22:15:22.465213 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.464560    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:22.477045 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.477011    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"]
Nov 02 22:15:22.481318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.481291    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894295-tjr6g"]
Nov 02 22:15:22.543221 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.543081    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:22.546298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.546268    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9330703a-25dd-4c73-a5b8-296354013803-kube-api-access-29r5v\" (UniqueName: \"kubernetes.io/projected/9330703a-25dd-4c73-a5b8-296354013803-kube-api-access-29r5v\") pod \"9330703a-25dd-4c73-a5b8-296354013803\" (UID: \"9330703a-25dd-4c73-a5b8-296354013803\") "
Nov 02 22:15:22.546433 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.546327    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9330703a-25dd-4c73-a5b8-296354013803-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330703a-25dd-4c73-a5b8-296354013803-maas-api-service-ca\") pod \"9330703a-25dd-4c73-a5b8-296354013803\" (UID: \"9330703a-25dd-4c73-a5b8-296354013803\") "
Nov 02 22:15:22.547059 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.547030    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9330703a-25dd-4c73-a5b8-296354013803-maas-api-service-ca" pod "9330703a-25dd-4c73-a5b8-296354013803" (UID: "9330703a-25dd-4c73-a5b8-296354013803"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:22.550208 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.549845    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9330703a-25dd-4c73-a5b8-296354013803-kube-api-access-29r5v" pod "9330703a-25dd-4c73-a5b8-296354013803" (UID: "9330703a-25dd-4c73-a5b8-296354013803"). InnerVolumeSpecName "kube-api-access-29r5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:22.550519 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.550496    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:22.574831 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.573863    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:22.648170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.647845    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dbc0a56-99f7-4467-8501-8a50b683cb6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc0a56-99f7-4467-8501-8a50b683cb6e-maas-api-service-ca\") pod \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\" (UID: \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\") "
Nov 02 22:15:22.648170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.647884    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbc0a56-99f7-4467-8501-8a50b683cb6e-kube-api-access-49gxd\" (UniqueName: \"kubernetes.io/projected/0dbc0a56-99f7-4467-8501-8a50b683cb6e-kube-api-access-49gxd\") pod \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\" (UID: \"0dbc0a56-99f7-4467-8501-8a50b683cb6e\") "
Nov 02 22:15:22.648170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.648027    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-maas-api-service-ca\") pod \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\" (UID: \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\") "
Nov 02 22:15:22.648170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.648078    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-kube-api-access-8chm2\" (UniqueName: \"kubernetes.io/projected/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-kube-api-access-8chm2\") pod \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\" (UID: \"b736b5d4-392b-4ee7-8757-dfe9b0b63c16\") "
Nov 02 22:15:22.648422 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.648239    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6cda15-3a4d-4450-b922-aa263f332786-kube-api-access-gnh25\" (UniqueName: \"kubernetes.io/projected/3e6cda15-3a4d-4450-b922-aa263f332786-kube-api-access-gnh25\") pod \"3e6cda15-3a4d-4450-b922-aa263f332786\" (UID: \"3e6cda15-3a4d-4450-b922-aa263f332786\") "
Nov 02 22:15:22.648422 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.648336    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6cda15-3a4d-4450-b922-aa263f332786-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cda15-3a4d-4450-b922-aa263f332786-maas-api-service-ca\") pod \"3e6cda15-3a4d-4450-b922-aa263f332786\" (UID: \"3e6cda15-3a4d-4450-b922-aa263f332786\") "
Nov 02 22:15:22.648422 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.648385    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbc0a56-99f7-4467-8501-8a50b683cb6e-maas-api-service-ca" pod "0dbc0a56-99f7-4467-8501-8a50b683cb6e" (UID: "0dbc0a56-99f7-4467-8501-8a50b683cb6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:22.648523 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.648489    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-maas-api-service-ca" pod "b736b5d4-392b-4ee7-8757-dfe9b0b63c16" (UID: "b736b5d4-392b-4ee7-8757-dfe9b0b63c16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:22.649068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.649003    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc0a56-99f7-4467-8501-8a50b683cb6e-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.649068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.649023    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.649068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.649033    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29r5v\" (UniqueName: \"kubernetes.io/projected/9330703a-25dd-4c73-a5b8-296354013803-kube-api-access-29r5v\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.649068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.649057    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9330703a-25dd-4c73-a5b8-296354013803-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.649375 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.649258    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6cda15-3a4d-4450-b922-aa263f332786-maas-api-service-ca" pod "3e6cda15-3a4d-4450-b922-aa263f332786" (UID: "3e6cda15-3a4d-4450-b922-aa263f332786"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:22.650956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.650930    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbc0a56-99f7-4467-8501-8a50b683cb6e-kube-api-access-49gxd" pod "0dbc0a56-99f7-4467-8501-8a50b683cb6e" (UID: "0dbc0a56-99f7-4467-8501-8a50b683cb6e"). InnerVolumeSpecName "kube-api-access-49gxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:22.651078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.650960    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6cda15-3a4d-4450-b922-aa263f332786-kube-api-access-gnh25" pod "3e6cda15-3a4d-4450-b922-aa263f332786" (UID: "3e6cda15-3a4d-4450-b922-aa263f332786"). InnerVolumeSpecName "kube-api-access-gnh25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:22.651294 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.651274    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-kube-api-access-8chm2" pod "b736b5d4-392b-4ee7-8757-dfe9b0b63c16" (UID: "b736b5d4-392b-4ee7-8757-dfe9b0b63c16"). InnerVolumeSpecName "kube-api-access-8chm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:22.653608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.653589    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:22.749719 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.749615    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-kube-api-access-vj7fd\" (UniqueName: \"kubernetes.io/projected/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-kube-api-access-vj7fd\") pod \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\" (UID: \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\") "
Nov 02 22:15:22.749719 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.749683    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-maas-api-service-ca\") pod \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\" (UID: \"e5a4167b-2636-40ef-842d-9ea3cc8c46ea\") "
Nov 02 22:15:22.750094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.750077    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8chm2\" (UniqueName: \"kubernetes.io/projected/b736b5d4-392b-4ee7-8757-dfe9b0b63c16-kube-api-access-8chm2\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.750203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.750083    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-maas-api-service-ca" pod "e5a4167b-2636-40ef-842d-9ea3cc8c46ea" (UID: "e5a4167b-2636-40ef-842d-9ea3cc8c46ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:22.750203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.750095    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnh25\" (UniqueName: \"kubernetes.io/projected/3e6cda15-3a4d-4450-b922-aa263f332786-kube-api-access-gnh25\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.750203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.750141    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cda15-3a4d-4450-b922-aa263f332786-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.750203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.750151    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49gxd\" (UniqueName: \"kubernetes.io/projected/0dbc0a56-99f7-4467-8501-8a50b683cb6e-kube-api-access-49gxd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.751816 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.751784    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-kube-api-access-vj7fd" pod "e5a4167b-2636-40ef-842d-9ea3cc8c46ea" (UID: "e5a4167b-2636-40ef-842d-9ea3cc8c46ea"). InnerVolumeSpecName "kube-api-access-vj7fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:22.852747 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.852702    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7fd\" (UniqueName: \"kubernetes.io/projected/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-kube-api-access-vj7fd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.852959 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.852779    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a4167b-2636-40ef-842d-9ea3cc8c46ea-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:22.896668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.896634    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"]
Nov 02 22:15:22.899919 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.899888    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894295-98lk6"]
Nov 02 22:15:22.938535 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.938497    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" path="/var/lib/kubelet/pods/204147df-3fd6-4c5c-b2b4-691e5c45d9bc/volumes"
Nov 02 22:15:22.939190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.939172    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" path="/var/lib/kubelet/pods/3b34b9e5-ff28-4e9f-82bd-1e002b2366a1/volumes"
Nov 02 22:15:22.939788 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.939771    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660176c3-ce07-49f1-af67-55c1d191a517" path="/var/lib/kubelet/pods/660176c3-ce07-49f1-af67-55c1d191a517/volumes"
Nov 02 22:15:22.940539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.940522    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" path="/var/lib/kubelet/pods/9ba9bd48-8697-47b6-9c1a-37c67b8cef4f/volumes"
Nov 02 22:15:22.940902 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.940890    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" path="/var/lib/kubelet/pods/a21ffde3-3d62-4a69-abfc-3fecce92eb86/volumes"
Nov 02 22:15:22.941305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:22.941292    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" path="/var/lib/kubelet/pods/be619aa4-7336-40a9-b5d4-1a5ba5d69129/volumes"
Nov 02 22:15:23.080989 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.080959    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"]
Nov 02 22:15:23.084693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.084665    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-7mprd"]
Nov 02 22:15:23.176374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.176333    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"
Nov 02 22:15:23.176374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.176345    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n" event={"ID":"3e6cda15-3a4d-4450-b922-aa263f332786","Type":"ContainerDied","Data":"17cb5a25a5712b6eee060da9849838e007d8818265f289253e12421630b43bd1"}
Nov 02 22:15:23.176927 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.176416    4528 scope.go:122] "RemoveContainer" containerID="0a2138080a4cac5a601826510451d96934bcbbb1d15f64cf90a9007af83d3cc6"
Nov 02 22:15:23.178427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.178407    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"
Nov 02 22:15:23.178549 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.178409    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl" event={"ID":"b736b5d4-392b-4ee7-8757-dfe9b0b63c16","Type":"ContainerDied","Data":"4629bb4166a9369e5c5ea031e27cfde4e9091ea954fdd0c8bdfdba35700fd244"}
Nov 02 22:15:23.181201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.181179    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"
Nov 02 22:15:23.181304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.181195    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4" event={"ID":"0dbc0a56-99f7-4467-8501-8a50b683cb6e","Type":"ContainerDied","Data":"280a47280d648ba4b0a6b43cf3a92299f55e503808c88ca11e88dd91217cdddd"}
Nov 02 22:15:23.183161 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.183132    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g" event={"ID":"e5a4167b-2636-40ef-842d-9ea3cc8c46ea","Type":"ContainerDied","Data":"11ae6eb943eda3647b1c2d1df85252cded538e72a9a81ffaef6ed08d02a074a7"}
Nov 02 22:15:23.183289 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.183171    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"
Nov 02 22:15:23.185411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.185389    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22" event={"ID":"9330703a-25dd-4c73-a5b8-296354013803","Type":"ContainerDied","Data":"32fc11a91ddb1c886e35b48bdb13727dfa9512a8b451ab7394139147dcb54ee4"}
Nov 02 22:15:23.185517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.185505    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"
Nov 02 22:15:23.189281 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.189262    4528 scope.go:122] "RemoveContainer" containerID="e66f58bf0dd3065599eab6a163df233e2f3b79aa5743d04636c12f092fccb412"
Nov 02 22:15:23.200492 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.200473    4528 scope.go:122] "RemoveContainer" containerID="f349d95c5f46571bf7326b172bbf7a8a5518871e78ae5c4e5daf4790f7494a8a"
Nov 02 22:15:23.245217 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.245188    4528 scope.go:122] "RemoveContainer" containerID="8daedfd0048c538cfc0206a680f1c886c2f22ea3ee06b91d8934648c68f59e01"
Nov 02 22:15:23.256985 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.256961    4528 scope.go:122] "RemoveContainer" containerID="e6cfb8010b736b86d604ea8af3e47c3e52e8cf3202ec9fdeb6b3367748442e1c"
Nov 02 22:15:23.771662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.771630    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"]
Nov 02 22:15:23.774901 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.774877    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72f279-29894295-7qw22"]
Nov 02 22:15:23.899906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.899873    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"]
Nov 02 22:15:23.927276 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:23.927241    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f765a-29894295-fck4n"]
Nov 02 22:15:24.029529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.029437    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"]
Nov 02 22:15:24.039893 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.039849    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c7e71f-29894295-sht9g"]
Nov 02 22:15:24.137468 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.137435    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"]
Nov 02 22:15:24.140407 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.140392    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894295-jw4z4"]
Nov 02 22:15:24.248354 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.248314    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"]
Nov 02 22:15:24.251798 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.251775    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894295-b4gxl"]
Nov 02 22:15:24.832096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.832056    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"]
Nov 02 22:15:24.832339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.832316    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy" containerID="cri-o://95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e" gracePeriod=30
Nov 02 22:15:24.977135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.973939    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" path="/var/lib/kubelet/pods/0dbc0a56-99f7-4467-8501-8a50b683cb6e/volumes"
Nov 02 22:15:24.977135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.974604    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" path="/var/lib/kubelet/pods/3e6cda15-3a4d-4450-b922-aa263f332786/volumes"
Nov 02 22:15:24.977135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.975165    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" path="/var/lib/kubelet/pods/507f7d40-be0c-4471-9821-04f1fb7480e1/volumes"
Nov 02 22:15:25.000131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.979520    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9330703a-25dd-4c73-a5b8-296354013803" path="/var/lib/kubelet/pods/9330703a-25dd-4c73-a5b8-296354013803/volumes"
Nov 02 22:15:25.000131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.980054    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" path="/var/lib/kubelet/pods/b736b5d4-392b-4ee7-8757-dfe9b0b63c16/volumes"
Nov 02 22:15:25.000131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:24.980563    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" path="/var/lib/kubelet/pods/e5a4167b-2636-40ef-842d-9ea3cc8c46ea/volumes"
Nov 02 22:15:30.105028 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.104946    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:15:30.187631 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.187590    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-credential-socket\") pod \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") "
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.187710    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-envoy\") pod \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") "
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.187758    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istiod-ca-cert\") pod \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") "
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.187784    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-token\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-token\") pod \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") "
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.187850    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-certs\") pod \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") "
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.187857    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-credential-socket" pod "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" (UID: "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.187879    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-kube-api-access-7clzm\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-kube-api-access-7clzm\") pod \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") "
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.187984    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-socket\") pod \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") "
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188068    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-podinfo\") pod \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") "
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188101    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-data\") pod \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\" (UID: \"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70\") "
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188380    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-socket" pod "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" (UID: "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188417    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istiod-ca-cert" pod "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" (UID: "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188628    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-data" pod "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" (UID: "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188680    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-certs" pod "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" (UID: "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188814    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188838    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188852    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188864    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.188878    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.190396    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-token" pod "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" (UID: "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.190478    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-envoy" pod "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" (UID: "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.190539    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-kube-api-access-7clzm" pod "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" (UID: "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70"). InnerVolumeSpecName "kube-api-access-7clzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:15:30.217167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.192001    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-podinfo" pod "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" (UID: "494fe0ee-eeb6-49a1-9d99-1a58e6edfe70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:15:30.273003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.272429    4528 generic.go:356] "Generic (PLEG): container finished" podID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerID="95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e" exitCode=0
Nov 02 22:15:30.273003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.272494    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" event={"ID":"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70","Type":"ContainerDied","Data":"95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e"}
Nov 02 22:15:30.273003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.272508    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"
Nov 02 22:15:30.273003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.272568    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx" event={"ID":"494fe0ee-eeb6-49a1-9d99-1a58e6edfe70","Type":"ContainerDied","Data":"ba045456a9bb4c47636956e36ab3f83a30eb67c00287a1876bae7f821615a33b"}
Nov 02 22:15:30.273003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.272597    4528 scope.go:122] "RemoveContainer" containerID="95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e"
Nov 02 22:15:30.288354 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.286926    4528 scope.go:122] "RemoveContainer" containerID="b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b"
Nov 02 22:15:30.290065 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.290029    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:30.290065 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.290065    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:30.290595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.290079    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:30.290595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.290092    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7clzm\" (UniqueName: \"kubernetes.io/projected/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70-kube-api-access-7clzm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:15:30.304852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.304824    4528 scope.go:122] "RemoveContainer" containerID="95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e"
Nov 02 22:15:30.305243 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:30.305221    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e\": container with ID starting with 95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e not found: ID does not exist" containerID="95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e"
Nov 02 22:15:30.305365 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.305251    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e"} err="failed to get container status \"95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e\": rpc error: code = NotFound desc = could not find container \"95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e\": container with ID starting with 95803d74e0ef1bd3b726de3f0938335b47923a1996d99f98a3e1c61ef83a639e not found: ID does not exist"
Nov 02 22:15:30.305365 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.305276    4528 scope.go:122] "RemoveContainer" containerID="b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b"
Nov 02 22:15:30.305540 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:30.305518    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b\": container with ID starting with b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b not found: ID does not exist" containerID="b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b"
Nov 02 22:15:30.305604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.305552    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b"} err="failed to get container status \"b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b\": rpc error: code = NotFound desc = could not find container \"b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b\": container with ID starting with b3299d7e8d423e0292f7a0c14bf897b8955885f62e89e1bb68f522a942d2001b not found: ID does not exist"
Nov 02 22:15:30.437907 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.437763    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"]
Nov 02 22:15:30.441050 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.441026    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-065be7-openshift-default-64b876f68f-krrvx"]
Nov 02 22:15:30.915832 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:30.915799    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" path="/var/lib/kubelet/pods/494fe0ee-eeb6-49a1-9d99-1a58e6edfe70/volumes"
Nov 02 22:15:31.282846 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:31.282817    4528 generic.go:356] "Generic (PLEG): container finished" podID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerID="696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab" exitCode=0
Nov 02 22:15:31.283257 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:31.282886    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" event={"ID":"50aa6c40-0fa9-430a-94ea-12b06cb46bd8","Type":"ContainerDied","Data":"696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab"}
Nov 02 22:15:31.283336 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:31.283293    4528 scope.go:122] "RemoveContainer" containerID="696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab"
Nov 02 22:15:31.284930 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:31.284904    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:15:31.285098 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:31.284998    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:15:31.285098 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:31.285037    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:15:32.291511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:32.291476    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" event={"ID":"50aa6c40-0fa9-430a-94ea-12b06cb46bd8","Type":"ContainerStarted","Data":"3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c"}
Nov 02 22:15:32.326134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:32.317230    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:15:32.620189 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:32.616479    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:15:32.626314 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:32.626273    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 22:15:32.626503 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:32.626358    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 22:15:33.626136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:33.624633    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 22:15:33.626136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:33.624703    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 22:15:34.626510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:34.626469    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 22:15:34.626960 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:34.626549    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 22:15:35.614302 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:35.614262    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 22:15:35.614601 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:35.614572    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 22:15:36.617241 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:36.617201    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 02 22:15:36.617850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:36.617814    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 02 22:15:37.616032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:37.616002    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:15:37.617246 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:15:37.617223    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:15:44.114991 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:15:44.114950    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:16:13.008617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:13.008583    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"]
Nov 02 22:16:13.114259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:13.114224    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"]
Nov 02 22:16:14.114607 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:16:14.114571    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:16:18.009049 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.008991    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph" podUID="9db93c8f-772f-42f0-8d7e-beabf47d06d9" containerName="payload-processing" containerID="cri-o://abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd" gracePeriod=25
Nov 02 22:16:18.115049 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.115008    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4" podUID="3afb3dbf-9f78-482c-b663-9efefcf18168" containerName="payload-pre-processing" containerID="cri-o://b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e" gracePeriod=25
Nov 02 22:16:18.298318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.298292    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:16:18.331731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.331697    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db93c8f-772f-42f0-8d7e-beabf47d06d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db93c8f-772f-42f0-8d7e-beabf47d06d9-plugins-config-volume\") pod \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\" (UID: \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\") "
Nov 02 22:16:18.331952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.331836    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db93c8f-772f-42f0-8d7e-beabf47d06d9-kube-api-access-d9hjs\" (UniqueName: \"kubernetes.io/projected/9db93c8f-772f-42f0-8d7e-beabf47d06d9-kube-api-access-d9hjs\") pod \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\" (UID: \"9db93c8f-772f-42f0-8d7e-beabf47d06d9\") "
Nov 02 22:16:18.332225 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.332198    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db93c8f-772f-42f0-8d7e-beabf47d06d9-plugins-config-volume" pod "9db93c8f-772f-42f0-8d7e-beabf47d06d9" (UID: "9db93c8f-772f-42f0-8d7e-beabf47d06d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:16:18.332361 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.332342    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db93c8f-772f-42f0-8d7e-beabf47d06d9-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:18.334342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.334307    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db93c8f-772f-42f0-8d7e-beabf47d06d9-kube-api-access-d9hjs" pod "9db93c8f-772f-42f0-8d7e-beabf47d06d9" (UID: "9db93c8f-772f-42f0-8d7e-beabf47d06d9"). InnerVolumeSpecName "kube-api-access-d9hjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:18.397363 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.397334    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:16:18.433661 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.433629    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afb3dbf-9f78-482c-b663-9efefcf18168-kube-api-access-g84zg\" (UniqueName: \"kubernetes.io/projected/3afb3dbf-9f78-482c-b663-9efefcf18168-kube-api-access-g84zg\") pod \"3afb3dbf-9f78-482c-b663-9efefcf18168\" (UID: \"3afb3dbf-9f78-482c-b663-9efefcf18168\") "
Nov 02 22:16:18.433881 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.433758    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afb3dbf-9f78-482c-b663-9efefcf18168-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afb3dbf-9f78-482c-b663-9efefcf18168-plugins-config-volume\") pod \"3afb3dbf-9f78-482c-b663-9efefcf18168\" (UID: \"3afb3dbf-9f78-482c-b663-9efefcf18168\") "
Nov 02 22:16:18.434198 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.434159    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afb3dbf-9f78-482c-b663-9efefcf18168-plugins-config-volume" pod "3afb3dbf-9f78-482c-b663-9efefcf18168" (UID: "3afb3dbf-9f78-482c-b663-9efefcf18168"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:16:18.434418 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.434401    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afb3dbf-9f78-482c-b663-9efefcf18168-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:18.434502 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.434419    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9hjs\" (UniqueName: \"kubernetes.io/projected/9db93c8f-772f-42f0-8d7e-beabf47d06d9-kube-api-access-d9hjs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:18.435950 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.435928    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afb3dbf-9f78-482c-b663-9efefcf18168-kube-api-access-g84zg" pod "3afb3dbf-9f78-482c-b663-9efefcf18168" (UID: "3afb3dbf-9f78-482c-b663-9efefcf18168"). InnerVolumeSpecName "kube-api-access-g84zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:18.535368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.535283    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g84zg\" (UniqueName: \"kubernetes.io/projected/3afb3dbf-9f78-482c-b663-9efefcf18168-kube-api-access-g84zg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:18.733949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.733917    4528 generic.go:356] "Generic (PLEG): container finished" podID="9db93c8f-772f-42f0-8d7e-beabf47d06d9" containerID="abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd" exitCode=0
Nov 02 22:16:18.734157 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.733959    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"
Nov 02 22:16:18.734157 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.733996    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph" event={"ID":"9db93c8f-772f-42f0-8d7e-beabf47d06d9","Type":"ContainerDied","Data":"abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd"}
Nov 02 22:16:18.734157 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.734055    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph" event={"ID":"9db93c8f-772f-42f0-8d7e-beabf47d06d9","Type":"ContainerDied","Data":"8a65caf66b122a9477f7e4c426c925926cf72f5e9951c5dca86ea0487a305009"}
Nov 02 22:16:18.734157 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.734081    4528 scope.go:122] "RemoveContainer" containerID="abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd"
Nov 02 22:16:18.735607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.735580    4528 generic.go:356] "Generic (PLEG): container finished" podID="3afb3dbf-9f78-482c-b663-9efefcf18168" containerID="b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e" exitCode=0
Nov 02 22:16:18.735734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.735622    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"
Nov 02 22:16:18.735734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.735701    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4" event={"ID":"3afb3dbf-9f78-482c-b663-9efefcf18168","Type":"ContainerDied","Data":"b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e"}
Nov 02 22:16:18.735856 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.735742    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4" event={"ID":"3afb3dbf-9f78-482c-b663-9efefcf18168","Type":"ContainerDied","Data":"3b9d5491e272ebdca568ffba39adb7a0660d2ce8ea5cb90815a37aeaf9c3597f"}
Nov 02 22:16:18.745103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.744924    4528 scope.go:122] "RemoveContainer" containerID="abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd"
Nov 02 22:16:18.745276 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:16:18.745251    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd\": container with ID starting with abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd not found: ID does not exist" containerID="abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd"
Nov 02 22:16:18.745338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.745286    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd"} err="failed to get container status \"abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd\": rpc error: code = NotFound desc = could not find container \"abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd\": container with ID starting with abb79b6de48260f9e66cbd6d280e1b5debf32d9a4f01489a1696b2a4bbced2dd not found: ID does not exist"
Nov 02 22:16:18.745338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.745307    4528 scope.go:122] "RemoveContainer" containerID="b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e"
Nov 02 22:16:18.754183 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.754161    4528 scope.go:122] "RemoveContainer" containerID="b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e"
Nov 02 22:16:18.754439 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:16:18.754415    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e\": container with ID starting with b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e not found: ID does not exist" containerID="b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e"
Nov 02 22:16:18.754529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.754445    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e"} err="failed to get container status \"b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e\": rpc error: code = NotFound desc = could not find container \"b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e\": container with ID starting with b93e4441c2409b8764efdc416c02853a11c64d7d5cc3b90c67893dbaeb8cd89e not found: ID does not exist"
Nov 02 22:16:18.900031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.899938    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"]
Nov 02 22:16:18.905393 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.905360    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72f279-645d65f547-wcwph"]
Nov 02 22:16:18.914740 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:18.914708    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db93c8f-772f-42f0-8d7e-beabf47d06d9" path="/var/lib/kubelet/pods/9db93c8f-772f-42f0-8d7e-beabf47d06d9/volumes"
Nov 02 22:16:19.013915 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:19.013879    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"]
Nov 02 22:16:19.016878 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:19.016852    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72f279-5c97d88cc9-4nsw4"]
Nov 02 22:16:20.914462 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:20.914430    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afb3dbf-9f78-482c-b663-9efefcf18168" path="/var/lib/kubelet/pods/3afb3dbf-9f78-482c-b663-9efefcf18168/volumes"
Nov 02 22:16:27.426135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:27.424510    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"]
Nov 02 22:16:27.519547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:27.519464    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"]
Nov 02 22:16:27.771036 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:27.770994    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:16:32.425276 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.425232    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7" podUID="5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764" containerName="payload-processing" containerID="cri-o://127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37" gracePeriod=25
Nov 02 22:16:32.520087 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.520017    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq" podUID="8a78a940-be05-41d1-808d-d7df4d3b076b" containerName="payload-pre-processing" containerID="cri-o://80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c" gracePeriod=25
Nov 02 22:16:32.800999 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.800969    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:16:32.832182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.832159    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:16:32.895571 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.895527    4528 generic.go:356] "Generic (PLEG): container finished" podID="5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764" containerID="127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37" exitCode=0
Nov 02 22:16:32.895732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.895576    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"
Nov 02 22:16:32.895732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.895613    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7" event={"ID":"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764","Type":"ContainerDied","Data":"127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37"}
Nov 02 22:16:32.895732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.895672    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7" event={"ID":"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764","Type":"ContainerDied","Data":"d0207f9a774cf1900e666a0e25640b45c0834014267f4f87b3e185253de5a23d"}
Nov 02 22:16:32.895732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.895701    4528 scope.go:122] "RemoveContainer" containerID="127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37"
Nov 02 22:16:32.897185 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.897160    4528 generic.go:356] "Generic (PLEG): container finished" podID="8a78a940-be05-41d1-808d-d7df4d3b076b" containerID="80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c" exitCode=0
Nov 02 22:16:32.897302 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.897216    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"
Nov 02 22:16:32.897302 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.897232    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq" event={"ID":"8a78a940-be05-41d1-808d-d7df4d3b076b","Type":"ContainerDied","Data":"80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c"}
Nov 02 22:16:32.897302 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.897279    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq" event={"ID":"8a78a940-be05-41d1-808d-d7df4d3b076b","Type":"ContainerDied","Data":"fd38ec22084a106a70fb3874efa0a4fd3afa9abd8e32be8ccfa198c1e13d71ca"}
Nov 02 22:16:32.906016 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.905992    4528 scope.go:122] "RemoveContainer" containerID="127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37"
Nov 02 22:16:32.906416 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:16:32.906397    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37\": container with ID starting with 127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37 not found: ID does not exist" containerID="127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37"
Nov 02 22:16:32.906511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.906425    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37"} err="failed to get container status \"127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37\": rpc error: code = NotFound desc = could not find container \"127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37\": container with ID starting with 127db4cdd6f9f0c56137600e031ecc4df7f4d99e754bd5d5951ad1003a175d37 not found: ID does not exist"
Nov 02 22:16:32.906511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.906442    4528 scope.go:122] "RemoveContainer" containerID="80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c"
Nov 02 22:16:32.922822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.922794    4528 scope.go:122] "RemoveContainer" containerID="80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c"
Nov 02 22:16:32.923295 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:16:32.923230    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c\": container with ID starting with 80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c not found: ID does not exist" containerID="80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c"
Nov 02 22:16:32.923295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.923270    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c"} err="failed to get container status \"80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c\": rpc error: code = NotFound desc = could not find container \"80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c\": container with ID starting with 80fa488a6edc39e9f770edc54894525c0281e0e4625a80da5bb3be1864869b8c not found: ID does not exist"
Nov 02 22:16:32.926633 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.926604    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-plugins-config-volume\") pod \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\" (UID: \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\") "
Nov 02 22:16:32.926777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.926702    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-kube-api-access-6q8rm\" (UniqueName: \"kubernetes.io/projected/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-kube-api-access-6q8rm\") pod \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\" (UID: \"5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764\") "
Nov 02 22:16:32.926777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.926744    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a78a940-be05-41d1-808d-d7df4d3b076b-kube-api-access-fvdgx\" (UniqueName: \"kubernetes.io/projected/8a78a940-be05-41d1-808d-d7df4d3b076b-kube-api-access-fvdgx\") pod \"8a78a940-be05-41d1-808d-d7df4d3b076b\" (UID: \"8a78a940-be05-41d1-808d-d7df4d3b076b\") "
Nov 02 22:16:32.926919 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.926780    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a78a940-be05-41d1-808d-d7df4d3b076b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a78a940-be05-41d1-808d-d7df4d3b076b-plugins-config-volume\") pod \"8a78a940-be05-41d1-808d-d7df4d3b076b\" (UID: \"8a78a940-be05-41d1-808d-d7df4d3b076b\") "
Nov 02 22:16:32.927148 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.927023    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-plugins-config-volume" pod "5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764" (UID: "5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:16:32.927293 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.927238    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a78a940-be05-41d1-808d-d7df4d3b076b-plugins-config-volume" pod "8a78a940-be05-41d1-808d-d7df4d3b076b" (UID: "8a78a940-be05-41d1-808d-d7df4d3b076b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:16:32.927466 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.927446    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:32.927515 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.927475    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a78a940-be05-41d1-808d-d7df4d3b076b-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:32.929398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.929368    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-kube-api-access-6q8rm" pod "5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764" (UID: "5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764"). InnerVolumeSpecName "kube-api-access-6q8rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:32.929610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:32.929586    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a78a940-be05-41d1-808d-d7df4d3b076b-kube-api-access-fvdgx" pod "8a78a940-be05-41d1-808d-d7df4d3b076b" (UID: "8a78a940-be05-41d1-808d-d7df4d3b076b"). InnerVolumeSpecName "kube-api-access-fvdgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:33.029029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:33.028997    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q8rm\" (UniqueName: \"kubernetes.io/projected/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764-kube-api-access-6q8rm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:33.029029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:33.029025    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvdgx\" (UniqueName: \"kubernetes.io/projected/8a78a940-be05-41d1-808d-d7df4d3b076b-kube-api-access-fvdgx\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:33.321589 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:33.321491    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"]
Nov 02 22:16:33.324510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:33.324482    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6b384-6bcff9d4dc-vsfcq"]
Nov 02 22:16:33.434808 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:33.434777    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"]
Nov 02 22:16:33.437775 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:33.437752    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6b384-7685ff7d9c-7mbt7"]
Nov 02 22:16:34.913620 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:34.913587    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764" path="/var/lib/kubelet/pods/5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764/volumes"
Nov 02 22:16:34.914025 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:34.914013    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a78a940-be05-41d1-808d-d7df4d3b076b" path="/var/lib/kubelet/pods/8a78a940-be05-41d1-808d-d7df4d3b076b/volumes"
Nov 02 22:16:36.144346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.144310    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"]
Nov 02 22:16:36.144972 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.144953    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" containerName="cleanup"
Nov 02 22:16:36.145014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.144974    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" containerName="cleanup"
Nov 02 22:16:36.145014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.144987    4528 cpu_manager.go:414] "containerMap: removing container" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerName="cleanup"
Nov 02 22:16:36.145014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.144992    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerName="cleanup"
Nov 02 22:16:36.145014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.144999    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerName="cleanup"
Nov 02 22:16:36.145014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145004    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerName="cleanup"
Nov 02 22:16:36.145014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145013    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145018    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145026    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145030    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145037    4528 cpu_manager.go:414] "containerMap: removing container" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145045    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145057    4528 cpu_manager.go:414] "containerMap: removing container" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145061    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145068    4528 cpu_manager.go:414] "containerMap: removing container" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145071    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145079    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9330703a-25dd-4c73-a5b8-296354013803" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145083    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9330703a-25dd-4c73-a5b8-296354013803" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145090    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8a78a940-be05-41d1-808d-d7df4d3b076b" containerName="payload-pre-processing"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145094    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a78a940-be05-41d1-808d-d7df4d3b076b" containerName="payload-pre-processing"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145101    4528 cpu_manager.go:414] "containerMap: removing container" podUID="660176c3-ce07-49f1-af67-55c1d191a517" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145105    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660176c3-ce07-49f1-af67-55c1d191a517" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145127    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145134    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145143    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145147    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145155    4528 cpu_manager.go:414] "containerMap: removing container" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145159    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145168    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145172    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145180    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145185    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145191    4528 cpu_manager.go:414] "containerMap: removing container" podUID="660176c3-ce07-49f1-af67-55c1d191a517" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145195    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660176c3-ce07-49f1-af67-55c1d191a517" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145207    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145215    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145223    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145227    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145234    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145238    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145245    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145249    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145254    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9330703a-25dd-4c73-a5b8-296354013803" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145258    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9330703a-25dd-4c73-a5b8-296354013803" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145264    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145268    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" containerName="cleanup"
Nov 02 22:16:36.145256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145280    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3afb3dbf-9f78-482c-b663-9efefcf18168" containerName="payload-pre-processing"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145288    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afb3dbf-9f78-482c-b663-9efefcf18168" containerName="payload-pre-processing"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145300    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764" containerName="payload-processing"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145304    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764" containerName="payload-processing"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145313    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145318    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145324    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145328    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145336    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145339    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145346    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145351    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145360    4528 cpu_manager.go:414] "containerMap: removing container" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145367    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145377    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145381    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145389    4528 cpu_manager.go:414] "containerMap: removing container" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145393    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145399    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145403    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145410    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9db93c8f-772f-42f0-8d7e-beabf47d06d9" containerName="payload-processing"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145414    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db93c8f-772f-42f0-8d7e-beabf47d06d9" containerName="payload-processing"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145428    4528 cpu_manager.go:414] "containerMap: removing container" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145433    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145528    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1ba9d7-47b4-4779-9ecc-a35d9ae5a764" containerName="payload-processing"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145543    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145552    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145557    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145564    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145569    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145575    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9330703a-25dd-4c73-a5b8-296354013803" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145580    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145585    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145589    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="660176c3-ce07-49f1-af67-55c1d191a517" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145596    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145602    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145607    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db93c8f-772f-42f0-8d7e-beabf47d06d9" containerName="payload-processing"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145613    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145620    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145625    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145631    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145636    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145642    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145647    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145652    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145657    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145663    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145668    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a78a940-be05-41d1-808d-d7df4d3b076b" containerName="payload-pre-processing"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145678    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145686    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145692    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145698    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b34b9e5-ff28-4e9f-82bd-1e002b2366a1" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145704    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="660176c3-ce07-49f1-af67-55c1d191a517" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145726    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145733    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a4167b-2636-40ef-842d-9ea3cc8c46ea" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145739    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9330703a-25dd-4c73-a5b8-296354013803" containerName="cleanup"
Nov 02 22:16:36.146497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.145745    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afb3dbf-9f78-482c-b663-9efefcf18168" containerName="payload-pre-processing"
Nov 02 22:16:36.150914 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.150896    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.199525 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.199493    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"]
Nov 02 22:16:36.246689 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.246660    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:16:36.246900 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.246705    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:16:36.246900 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.246740    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:16:36.253676 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.253656    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:16:36.268852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.268824    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7jhx\" (UniqueName: \"kubernetes.io/projected/ca2740c5-28e5-4c87-8457-aff181457bb5-kube-api-access-f7jhx\") pod \"maas-api-revoke-sub-599a7646-cjxg2\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.269048 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.268875    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-599a7646-cjxg2\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.269048 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.268941    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-599a7646-cjxg2\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.269463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.269445    4528 scope.go:122] "RemoveContainer" containerID="faa7889002a9396e406f60b812020dfce3722569f2490291ed1485898801225f"
Nov 02 22:16:36.279068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.279049    4528 scope.go:122] "RemoveContainer" containerID="cac116d4761883d7a008d85a04188e543773a8d9f2ef1beb7e2e04f0f5e92671"
Nov 02 22:16:36.287838 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.287809    4528 scope.go:122] "RemoveContainer" containerID="bb9bfca4a0fa8b4df909b264ae88818e709edebac0a4818f7ab4dce998e7ddbd"
Nov 02 22:16:36.370541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.370496    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-599a7646-cjxg2\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.370722 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.370674    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jhx\" (UniqueName: \"kubernetes.io/projected/ca2740c5-28e5-4c87-8457-aff181457bb5-kube-api-access-f7jhx\") pod \"maas-api-revoke-sub-599a7646-cjxg2\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.370768 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.370723    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-599a7646-cjxg2\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.371201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.371179    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-599a7646-cjxg2\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.373070 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.373049    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-599a7646-cjxg2\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.427492 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.427412    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7jhx\" (UniqueName: \"kubernetes.io/projected/ca2740c5-28e5-4c87-8457-aff181457bb5-kube-api-access-f7jhx\") pod \"maas-api-revoke-sub-599a7646-cjxg2\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.464912 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.464877    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:36.837090 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:16:36.837061    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2740c5_28e5_4c87_8457_aff181457bb5.slice/crio-880c18f3d3aca67cd88c84eb6da7ad614051b8903f5b87752474f3da6d1405db WatchSource:0}: Error finding container 880c18f3d3aca67cd88c84eb6da7ad614051b8903f5b87752474f3da6d1405db: Status 404 returned error can't find the container with id 880c18f3d3aca67cd88c84eb6da7ad614051b8903f5b87752474f3da6d1405db
Nov 02 22:16:36.840543 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.840521    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"]
Nov 02 22:16:36.924403 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:36.924368    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2" event={"ID":"ca2740c5-28e5-4c87-8457-aff181457bb5","Type":"ContainerStarted","Data":"880c18f3d3aca67cd88c84eb6da7ad614051b8903f5b87752474f3da6d1405db"}
Nov 02 22:16:37.931787 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:37.931752    4528 generic.go:356] "Generic (PLEG): container finished" podID="ca2740c5-28e5-4c87-8457-aff181457bb5" containerID="eb309e52f4880ed340c2f2a9b343cdc5fe5f98e3d0bbb0799a095d182e93c8da" exitCode=0
Nov 02 22:16:37.932222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:37.931832    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2" event={"ID":"ca2740c5-28e5-4c87-8457-aff181457bb5","Type":"ContainerDied","Data":"eb309e52f4880ed340c2f2a9b343cdc5fe5f98e3d0bbb0799a095d182e93c8da"}
Nov 02 22:16:39.083138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.083101    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:39.214568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.214488    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-xks-ca\") pod \"ca2740c5-28e5-4c87-8457-aff181457bb5\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") "
Nov 02 22:16:39.214737 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.214577    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2740c5-28e5-4c87-8457-aff181457bb5-kube-api-access-f7jhx\" (UniqueName: \"kubernetes.io/projected/ca2740c5-28e5-4c87-8457-aff181457bb5-kube-api-access-f7jhx\") pod \"ca2740c5-28e5-4c87-8457-aff181457bb5\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") "
Nov 02 22:16:39.214737 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.214609    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-openshift-ca\") pod \"ca2740c5-28e5-4c87-8457-aff181457bb5\" (UID: \"ca2740c5-28e5-4c87-8457-aff181457bb5\") "
Nov 02 22:16:39.215138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.215084    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-openshift-ca" pod "ca2740c5-28e5-4c87-8457-aff181457bb5" (UID: "ca2740c5-28e5-4c87-8457-aff181457bb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:16:39.216903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.216870    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2740c5-28e5-4c87-8457-aff181457bb5-kube-api-access-f7jhx" pod "ca2740c5-28e5-4c87-8457-aff181457bb5" (UID: "ca2740c5-28e5-4c87-8457-aff181457bb5"). InnerVolumeSpecName "kube-api-access-f7jhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:16:39.217027 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.216950    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-xks-ca" pod "ca2740c5-28e5-4c87-8457-aff181457bb5" (UID: "ca2740c5-28e5-4c87-8457-aff181457bb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:16:39.316083 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.316043    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:39.316083 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.316077    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7jhx\" (UniqueName: \"kubernetes.io/projected/ca2740c5-28e5-4c87-8457-aff181457bb5-kube-api-access-f7jhx\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:39.316083 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.316087    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2740c5-28e5-4c87-8457-aff181457bb5-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:16:39.947947 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.947911    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"
Nov 02 22:16:39.947947 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.947938    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2" event={"ID":"ca2740c5-28e5-4c87-8457-aff181457bb5","Type":"ContainerDied","Data":"880c18f3d3aca67cd88c84eb6da7ad614051b8903f5b87752474f3da6d1405db"}
Nov 02 22:16:39.948234 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:16:39.947989    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="880c18f3d3aca67cd88c84eb6da7ad614051b8903f5b87752474f3da6d1405db"
Nov 02 22:16:44.115458 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:16:44.115412    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:17:14.113792 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:17:14.113755    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:17:34.291718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:34.291684    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"]
Nov 02 22:17:34.292261 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:34.291897    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy" containerID="cri-o://154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be" gracePeriod=30
Nov 02 22:17:36.419973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:36.419931    4528 patch_prober.go:29] interesting pod/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:17:36.420369 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:36.420008    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:17:39.566080 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.566018    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:17:39.687187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687152    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-socket\") pod \"313de020-6c5e-455c-b7a0-d1a15228e8a1\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") "
Nov 02 22:17:39.687385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687205    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-credential-socket\") pod \"313de020-6c5e-455c-b7a0-d1a15228e8a1\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") "
Nov 02 22:17:39.687385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687232    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-podinfo\") pod \"313de020-6c5e-455c-b7a0-d1a15228e8a1\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") "
Nov 02 22:17:39.687385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687313    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-data\") pod \"313de020-6c5e-455c-b7a0-d1a15228e8a1\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") "
Nov 02 22:17:39.687385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687371    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-kube-api-access-ctmng\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-kube-api-access-ctmng\") pod \"313de020-6c5e-455c-b7a0-d1a15228e8a1\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") "
Nov 02 22:17:39.687615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687402    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313de020-6c5e-455c-b7a0-d1a15228e8a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313de020-6c5e-455c-b7a0-d1a15228e8a1-istiod-ca-cert\") pod \"313de020-6c5e-455c-b7a0-d1a15228e8a1\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") "
Nov 02 22:17:39.687615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687443    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-envoy\") pod \"313de020-6c5e-455c-b7a0-d1a15228e8a1\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") "
Nov 02 22:17:39.687615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687457    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-socket" pod "313de020-6c5e-455c-b7a0-d1a15228e8a1" (UID: "313de020-6c5e-455c-b7a0-d1a15228e8a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:39.687615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687562    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-credential-socket" pod "313de020-6c5e-455c-b7a0-d1a15228e8a1" (UID: "313de020-6c5e-455c-b7a0-d1a15228e8a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:39.687615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687567    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-certs\") pod \"313de020-6c5e-455c-b7a0-d1a15228e8a1\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") "
Nov 02 22:17:39.687895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687674    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-token\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-token\") pod \"313de020-6c5e-455c-b7a0-d1a15228e8a1\" (UID: \"313de020-6c5e-455c-b7a0-d1a15228e8a1\") "
Nov 02 22:17:39.687895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687785    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-certs" pod "313de020-6c5e-455c-b7a0-d1a15228e8a1" (UID: "313de020-6c5e-455c-b7a0-d1a15228e8a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:39.687895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687822    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313de020-6c5e-455c-b7a0-d1a15228e8a1-istiod-ca-cert" pod "313de020-6c5e-455c-b7a0-d1a15228e8a1" (UID: "313de020-6c5e-455c-b7a0-d1a15228e8a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:17:39.688065 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.687958    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-data" pod "313de020-6c5e-455c-b7a0-d1a15228e8a1" (UID: "313de020-6c5e-455c-b7a0-d1a15228e8a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:39.688290 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.688266    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:39.688398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.688293    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:39.688398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.688308    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:39.688398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.688321    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:39.688398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.688334    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313de020-6c5e-455c-b7a0-d1a15228e8a1-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:39.689761 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.689732    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-podinfo" pod "313de020-6c5e-455c-b7a0-d1a15228e8a1" (UID: "313de020-6c5e-455c-b7a0-d1a15228e8a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:17:39.689883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.689838    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-kube-api-access-ctmng" pod "313de020-6c5e-455c-b7a0-d1a15228e8a1" (UID: "313de020-6c5e-455c-b7a0-d1a15228e8a1"). InnerVolumeSpecName "kube-api-access-ctmng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:17:39.689883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.689858    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-envoy" pod "313de020-6c5e-455c-b7a0-d1a15228e8a1" (UID: "313de020-6c5e-455c-b7a0-d1a15228e8a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:39.690190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.690158    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-token" pod "313de020-6c5e-455c-b7a0-d1a15228e8a1" (UID: "313de020-6c5e-455c-b7a0-d1a15228e8a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:17:39.789766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.789734    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:39.789766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.789762    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:39.789766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.789772    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmng\" (UniqueName: \"kubernetes.io/projected/313de020-6c5e-455c-b7a0-d1a15228e8a1-kube-api-access-ctmng\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:39.790083 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.789783    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313de020-6c5e-455c-b7a0-d1a15228e8a1-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:39.930581 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.930488    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"]
Nov 02 22:17:39.931172 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931153    4528 cpu_manager.go:414] "containerMap: removing container" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerName="cleanup"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931173    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerName="cleanup"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931200    4528 cpu_manager.go:414] "containerMap: removing container" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931207    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931220    4528 cpu_manager.go:414] "containerMap: removing container" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931227    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494fe0ee-eeb6-49a1-9d99-1a58e6edfe70" containerName="istio-proxy"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931237    4528 cpu_manager.go:414] "containerMap: removing container" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931244    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931258    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ca2740c5-28e5-4c87-8457-aff181457bb5" containerName="revoke-keys"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931264    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2740c5-28e5-4c87-8457-aff181457bb5" containerName="revoke-keys"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931278    4528 cpu_manager.go:414] "containerMap: removing container" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerName="cleanup"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931284    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerName="cleanup"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931300    4528 cpu_manager.go:414] "containerMap: removing container" podUID="660176c3-ce07-49f1-af67-55c1d191a517" containerName="cleanup"
Nov 02 22:17:39.931303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931307    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660176c3-ce07-49f1-af67-55c1d191a517" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931323    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9330703a-25dd-4c73-a5b8-296354013803" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931330    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9330703a-25dd-4c73-a5b8-296354013803" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931356    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931363    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbc0a56-99f7-4467-8501-8a50b683cb6e" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931379    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931387    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931533    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="660176c3-ce07-49f1-af67-55c1d191a517" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931549    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9330703a-25dd-4c73-a5b8-296354013803" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931558    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="507f7d40-be0c-4471-9821-04f1fb7480e1" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931570    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="be619aa4-7336-40a9-b5d4-1a5ba5d69129" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931581    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba9bd48-8697-47b6-9c1a-37c67b8cef4f" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931590    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6cda15-3a4d-4450-b922-aa263f332786" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931600    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="204147df-3fd6-4c5c-b2b4-691e5c45d9bc" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931609    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21ffde3-3d62-4a69-abfc-3fecce92eb86" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931618    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b736b5d4-392b-4ee7-8757-dfe9b0b63c16" containerName="cleanup"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931626    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931633    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerName="istio-proxy"
Nov 02 22:17:39.931965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.931645    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2740c5-28e5-4c87-8457-aff181457bb5" containerName="revoke-keys"
Nov 02 22:17:39.935728 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.935709    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:39.987678 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.987640    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"]
Nov 02 22:17:39.988451 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.988424    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:17:39.988451 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.988444    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:17:39.988647 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.988437    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:17:39.988647 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:39.988595    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:17:40.093813 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.093777    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89fa6019-85fch\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.094038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.093958    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kzh2\" (UniqueName: \"kubernetes.io/projected/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-kube-api-access-5kzh2\") pod \"maas-api-revoke-sub-89fa6019-85fch\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.094038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.094016    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89fa6019-85fch\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.195510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.195426    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kzh2\" (UniqueName: \"kubernetes.io/projected/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-kube-api-access-5kzh2\") pod \"maas-api-revoke-sub-89fa6019-85fch\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.195510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.195470    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89fa6019-85fch\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.195757 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.195587    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89fa6019-85fch\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.196142 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.196103    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89fa6019-85fch\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.197843 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.197823    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89fa6019-85fch\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.345078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.344890    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kzh2\" (UniqueName: \"kubernetes.io/projected/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-kube-api-access-5kzh2\") pod \"maas-api-revoke-sub-89fa6019-85fch\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.444468 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.444430    4528 generic.go:356] "Generic (PLEG): container finished" podID="313de020-6c5e-455c-b7a0-d1a15228e8a1" containerID="154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be" exitCode=0
Nov 02 22:17:40.445359 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.445334    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"
Nov 02 22:17:40.475146 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.475009    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" event={"ID":"313de020-6c5e-455c-b7a0-d1a15228e8a1","Type":"ContainerDied","Data":"154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be"}
Nov 02 22:17:40.475146 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.475099    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc" event={"ID":"313de020-6c5e-455c-b7a0-d1a15228e8a1","Type":"ContainerDied","Data":"2fe9943f631ac89b08a3a8e29ca5cc19321929436038cc5e9e09a4d348d4de55"}
Nov 02 22:17:40.475386 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.475147    4528 scope.go:122] "RemoveContainer" containerID="154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be"
Nov 02 22:17:40.493952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.493793    4528 scope.go:122] "RemoveContainer" containerID="52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538"
Nov 02 22:17:40.503585 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.503564    4528 scope.go:122] "RemoveContainer" containerID="154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be"
Nov 02 22:17:40.503880 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:17:40.503855    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be\": container with ID starting with 154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be not found: ID does not exist" containerID="154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be"
Nov 02 22:17:40.503925 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.503890    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be"} err="failed to get container status \"154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be\": rpc error: code = NotFound desc = could not find container \"154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be\": container with ID starting with 154e1ff5f33edfc883103d59eeb4e87aac03b2622dae5c434996bcc58a7d19be not found: ID does not exist"
Nov 02 22:17:40.503925 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.503909    4528 scope.go:122] "RemoveContainer" containerID="52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538"
Nov 02 22:17:40.504182 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:17:40.504165    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538\": container with ID starting with 52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538 not found: ID does not exist" containerID="52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538"
Nov 02 22:17:40.504252 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.504186    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538"} err="failed to get container status \"52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538\": rpc error: code = NotFound desc = could not find container \"52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538\": container with ID starting with 52e51fbb1db828735dfaced81f27d305c921eef910ec1c09f0141de2725cf538 not found: ID does not exist"
Nov 02 22:17:40.549358 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.549330    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:40.647435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.647372    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"]
Nov 02 22:17:40.687058 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.687002    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-72f279-openshift-default-787b5c7c-2wtcc"]
Nov 02 22:17:40.914621 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.914590    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313de020-6c5e-455c-b7a0-d1a15228e8a1" path="/var/lib/kubelet/pods/313de020-6c5e-455c-b7a0-d1a15228e8a1/volumes"
Nov 02 22:17:40.920693 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:17:40.920665    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3d3f57_3ef8_4cfb_9aee_1ac1ab0ce6df.slice/crio-303c071a000d9b03551c4bdf2ffee1acc890e1ab4222dabb068a6e09a1ee8b77 WatchSource:0}: Error finding container 303c071a000d9b03551c4bdf2ffee1acc890e1ab4222dabb068a6e09a1ee8b77: Status 404 returned error can't find the container with id 303c071a000d9b03551c4bdf2ffee1acc890e1ab4222dabb068a6e09a1ee8b77
Nov 02 22:17:40.923838 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:40.923815    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"]
Nov 02 22:17:41.455267 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:41.455228    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch" event={"ID":"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df","Type":"ContainerStarted","Data":"e1ae8c640321bc305ed37f8e2385d730201eb09a02632dd9aec7257735a86ba8"}
Nov 02 22:17:41.455459 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:41.455323    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch" event={"ID":"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df","Type":"ContainerStarted","Data":"303c071a000d9b03551c4bdf2ffee1acc890e1ab4222dabb068a6e09a1ee8b77"}
Nov 02 22:17:41.571360 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:41.571307    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch" podStartSLOduration=2.571291937 podStartE2EDuration="2.571291937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:41.568638886 +0000 UTC m=+4148.832175205" watchObservedRunningTime="2026-11-02 22:17:41.571291937 +0000 UTC m=+4148.834828253"
Nov 02 22:17:42.462719 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:42.462689    4528 generic.go:356] "Generic (PLEG): container finished" podID="ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df" containerID="e1ae8c640321bc305ed37f8e2385d730201eb09a02632dd9aec7257735a86ba8" exitCode=0
Nov 02 22:17:42.463097 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:42.462741    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch" event={"ID":"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df","Type":"ContainerDied","Data":"e1ae8c640321bc305ed37f8e2385d730201eb09a02632dd9aec7257735a86ba8"}
Nov 02 22:17:43.628139 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.628096    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:43.730269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.730166    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-kube-api-access-5kzh2\" (UniqueName: \"kubernetes.io/projected/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-kube-api-access-5kzh2\") pod \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") "
Nov 02 22:17:43.730269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.730241    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-openshift-ca\") pod \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") "
Nov 02 22:17:43.730525 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.730285    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-xks-ca\") pod \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\" (UID: \"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df\") "
Nov 02 22:17:43.730666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.730644    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-openshift-ca" pod "ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df" (UID: "ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:17:43.732355 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.732331    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-kube-api-access-5kzh2" pod "ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df" (UID: "ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df"). InnerVolumeSpecName "kube-api-access-5kzh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:17:43.732435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.732339    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-xks-ca" pod "ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df" (UID: "ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:17:43.831955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.831864    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kzh2\" (UniqueName: \"kubernetes.io/projected/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-kube-api-access-5kzh2\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:43.831955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.831907    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:43.831955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:43.831922    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:44.113886 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:17:44.113849    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:17:44.476176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:44.476072    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"
Nov 02 22:17:44.476176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:44.476098    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch" event={"ID":"ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df","Type":"ContainerDied","Data":"303c071a000d9b03551c4bdf2ffee1acc890e1ab4222dabb068a6e09a1ee8b77"}
Nov 02 22:17:44.476176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:44.476159    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="303c071a000d9b03551c4bdf2ffee1acc890e1ab4222dabb068a6e09a1ee8b77"
Nov 02 22:17:46.432953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:46.432904    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:17:48.579143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.578165    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"]
Nov 02 22:17:48.579718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.579156    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df" containerName="revoke-keys"
Nov 02 22:17:48.579718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.579182    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df" containerName="revoke-keys"
Nov 02 22:17:48.579718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.579360    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df" containerName="revoke-keys"
Nov 02 22:17:48.651131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.649925    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"]
Nov 02 22:17:48.651131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.650055    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:17:48.702712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.702670    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cf2a53\""
Nov 02 22:17:48.702862 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.702746    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cf2a53-dockercfg-gm5dv\""
Nov 02 22:17:48.704276 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.704254    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d832b07-2793-4c03-a52b-07f295cf3821-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj\" (UID: \"6d832b07-2793-4c03-a52b-07f295cf3821\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:17:48.704414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.704393    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6cc\" (UniqueName: \"kubernetes.io/projected/6d832b07-2793-4c03-a52b-07f295cf3821-kube-api-access-bc6cc\") pod \"payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj\" (UID: \"6d832b07-2793-4c03-a52b-07f295cf3821\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:17:48.806139 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.806087    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d832b07-2793-4c03-a52b-07f295cf3821-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj\" (UID: \"6d832b07-2793-4c03-a52b-07f295cf3821\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:17:48.806337 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.806277    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6cc\" (UniqueName: \"kubernetes.io/projected/6d832b07-2793-4c03-a52b-07f295cf3821-kube-api-access-bc6cc\") pod \"payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj\" (UID: \"6d832b07-2793-4c03-a52b-07f295cf3821\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:17:48.806854 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.806836    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d832b07-2793-4c03-a52b-07f295cf3821-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj\" (UID: \"6d832b07-2793-4c03-a52b-07f295cf3821\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:17:48.863172 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.863087    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6cc\" (UniqueName: \"kubernetes.io/projected/6d832b07-2793-4c03-a52b-07f295cf3821-kube-api-access-bc6cc\") pod \"payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj\" (UID: \"6d832b07-2793-4c03-a52b-07f295cf3821\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:17:48.975904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:48.975866    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:17:49.355539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:49.355503    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"]
Nov 02 22:17:49.512155 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:49.512099    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj" event={"ID":"6d832b07-2793-4c03-a52b-07f295cf3821","Type":"ContainerStarted","Data":"e90174d26661a9d9637f71fbfbbc074a2505be71986a3906c82f7d1de6e6152a"}
Nov 02 22:17:49.512332 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:49.512173    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj" event={"ID":"6d832b07-2793-4c03-a52b-07f295cf3821","Type":"ContainerStarted","Data":"6113c56ad28dd3cd04e4940f9aceb68bb6fa02c9388161b5dc36f87cdad8439f"}
Nov 02 22:17:49.512332 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:49.512227    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:17:49.630539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:49.630437    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj" podStartSLOduration=1.630423427 podStartE2EDuration="1.630423427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:49.628549893 +0000 UTC m=+4156.892086223" watchObservedRunningTime="2026-11-02 22:17:49.630423427 +0000 UTC m=+4156.893959744"
Nov 02 22:17:49.702439 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:49.702399    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"]
Nov 02 22:17:49.702664 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:49.702638    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy" containerID="cri-o://3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c" gracePeriod=30
Nov 02 22:17:52.618320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:52.618284    4528 patch_prober.go:29] interesting pod/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:17:52.618707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:52.618340    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:17:55.078070 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.078045    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:17:55.213367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213337    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-socket\") pod \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") "
Nov 02 22:17:55.213523 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213472    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-credential-socket\") pod \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") "
Nov 02 22:17:55.213523 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213498    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-certs\") pod \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") "
Nov 02 22:17:55.213622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213555    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istiod-ca-cert\") pod \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") "
Nov 02 22:17:55.213622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213566    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-socket" pod "50aa6c40-0fa9-430a-94ea-12b06cb46bd8" (UID: "50aa6c40-0fa9-430a-94ea-12b06cb46bd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:55.213622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213605    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-podinfo\") pod \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") "
Nov 02 22:17:55.213802 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213722    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-token\") pod \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") "
Nov 02 22:17:55.213802 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213767    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-credential-socket" pod "50aa6c40-0fa9-430a-94ea-12b06cb46bd8" (UID: "50aa6c40-0fa9-430a-94ea-12b06cb46bd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:55.213802 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213776    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-data\") pod \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") "
Nov 02 22:17:55.213977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213813    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-certs" pod "50aa6c40-0fa9-430a-94ea-12b06cb46bd8" (UID: "50aa6c40-0fa9-430a-94ea-12b06cb46bd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:55.213977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213836    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-envoy\") pod \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") "
Nov 02 22:17:55.213977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213873    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istiod-ca-cert" pod "50aa6c40-0fa9-430a-94ea-12b06cb46bd8" (UID: "50aa6c40-0fa9-430a-94ea-12b06cb46bd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:17:55.213977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213943    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-kube-api-access-p8cf2\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-kube-api-access-p8cf2\") pod \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\" (UID: \"50aa6c40-0fa9-430a-94ea-12b06cb46bd8\") "
Nov 02 22:17:55.214280 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.213997    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-data" pod "50aa6c40-0fa9-430a-94ea-12b06cb46bd8" (UID: "50aa6c40-0fa9-430a-94ea-12b06cb46bd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:55.214557 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.214525    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:55.214557 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.214553    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:55.214734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.214568    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:55.214734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.214582    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:55.214734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.214597    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:55.215867 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.215838    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-token" pod "50aa6c40-0fa9-430a-94ea-12b06cb46bd8" (UID: "50aa6c40-0fa9-430a-94ea-12b06cb46bd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:17:55.215867 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.215845    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-podinfo" pod "50aa6c40-0fa9-430a-94ea-12b06cb46bd8" (UID: "50aa6c40-0fa9-430a-94ea-12b06cb46bd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:17:55.216021 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.215957    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-kube-api-access-p8cf2" pod "50aa6c40-0fa9-430a-94ea-12b06cb46bd8" (UID: "50aa6c40-0fa9-430a-94ea-12b06cb46bd8"). InnerVolumeSpecName "kube-api-access-p8cf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:17:55.216021 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.216005    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-envoy" pod "50aa6c40-0fa9-430a-94ea-12b06cb46bd8" (UID: "50aa6c40-0fa9-430a-94ea-12b06cb46bd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:17:55.315559 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.315478    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:55.315559 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.315510    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:55.315559 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.315520    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:55.315559 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.315528    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8cf2\" (UniqueName: \"kubernetes.io/projected/50aa6c40-0fa9-430a-94ea-12b06cb46bd8-kube-api-access-p8cf2\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:17:55.559672 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.559633    4528 generic.go:356] "Generic (PLEG): container finished" podID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerID="3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c" exitCode=0
Nov 02 22:17:55.559841 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.559691    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"
Nov 02 22:17:55.559841 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.559692    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" event={"ID":"50aa6c40-0fa9-430a-94ea-12b06cb46bd8","Type":"ContainerDied","Data":"3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c"}
Nov 02 22:17:55.559987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.559840    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb" event={"ID":"50aa6c40-0fa9-430a-94ea-12b06cb46bd8","Type":"ContainerDied","Data":"6409b9d28376dedbc0b39c9fc64ba5e7f2c2e5688f79be1d10ec5f28945e8501"}
Nov 02 22:17:55.559987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.559870    4528 scope.go:122] "RemoveContainer" containerID="3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c"
Nov 02 22:17:55.570808 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.570786    4528 scope.go:122] "RemoveContainer" containerID="696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab"
Nov 02 22:17:55.579691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.579671    4528 scope.go:122] "RemoveContainer" containerID="3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c"
Nov 02 22:17:55.579945 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:17:55.579926    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c\": container with ID starting with 3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c not found: ID does not exist" containerID="3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c"
Nov 02 22:17:55.580014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.579953    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c"} err="failed to get container status \"3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c\": rpc error: code = NotFound desc = could not find container \"3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c\": container with ID starting with 3030dfc7dab77647044e366182ceaef8cb7308fb3fe13e01ceeebda20eb6c83c not found: ID does not exist"
Nov 02 22:17:55.580014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.579971    4528 scope.go:122] "RemoveContainer" containerID="696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab"
Nov 02 22:17:55.580273 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:17:55.580251    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab\": container with ID starting with 696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab not found: ID does not exist" containerID="696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab"
Nov 02 22:17:55.580336 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.580279    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab"} err="failed to get container status \"696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab\": rpc error: code = NotFound desc = could not find container \"696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab\": container with ID starting with 696a2f8d06686565dae149481b95cdce21595fe3a136d858eca2b781e9c3deab not found: ID does not exist"
Nov 02 22:17:55.726066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.726030    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"]
Nov 02 22:17:55.776654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:55.776617    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6b384-openshift-default-7689469749-6c7cb"]
Nov 02 22:17:56.914616 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:17:56.914581    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" path="/var/lib/kubelet/pods/50aa6c40-0fa9-430a-94ea-12b06cb46bd8/volumes"
Nov 02 22:18:00.523094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.523061    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:18:00.571699 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.571659    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"]
Nov 02 22:18:00.572298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.572283    4528 cpu_manager.go:414] "containerMap: removing container" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy"
Nov 02 22:18:00.572387 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.572299    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy"
Nov 02 22:18:00.572427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.572409    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy"
Nov 02 22:18:00.572427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.572422    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy"
Nov 02 22:18:00.577463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.577444    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:00.629491 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.629457    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"]
Nov 02 22:18:00.630426 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.630391    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-db69f7-dockercfg-8449q\""
Nov 02 22:18:00.630655 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.630633    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-db69f7\""
Nov 02 22:18:00.752153 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.749438    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5\" (UID: \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:00.752153 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.749631    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66jrx\" (UniqueName: \"kubernetes.io/projected/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-kube-api-access-66jrx\") pod \"payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5\" (UID: \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:00.769188 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.769149    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"]
Nov 02 22:18:00.769829 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.769808    4528 cpu_manager.go:414] "containerMap: removing container" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy"
Nov 02 22:18:00.769949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.769831    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50aa6c40-0fa9-430a-94ea-12b06cb46bd8" containerName="istio-proxy"
Nov 02 22:18:00.773611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.773550    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:00.818075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.818033    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"]
Nov 02 22:18:00.850788 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.850756    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5\" (UID: \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:00.850991 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.850821    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l\" (UID: \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:00.851072 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.850992    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66jrx\" (UniqueName: \"kubernetes.io/projected/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-kube-api-access-66jrx\") pod \"payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5\" (UID: \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:00.851161 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.851074    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lq8q\" (UniqueName: \"kubernetes.io/projected/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-kube-api-access-9lq8q\") pod \"payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l\" (UID: \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:00.851506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.851486    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5\" (UID: \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:00.908230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.908191    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66jrx\" (UniqueName: \"kubernetes.io/projected/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-kube-api-access-66jrx\") pod \"payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5\" (UID: \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:00.952425 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.952389    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lq8q\" (UniqueName: \"kubernetes.io/projected/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-kube-api-access-9lq8q\") pod \"payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l\" (UID: \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:00.952608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.952587    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l\" (UID: \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:00.953402 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:00.953380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l\" (UID: \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:01.010212 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.010171    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lq8q\" (UniqueName: \"kubernetes.io/projected/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-kube-api-access-9lq8q\") pod \"payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l\" (UID: \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:01.090780 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.090680    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:01.191806 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.191769    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:01.460253 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:18:01.460222    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d233018_a0a5_4914_8b5e_fd6a5bfef1fd.slice/crio-0efed9dd9d945d2f29991ccf98dcf428400a86faa2023a66bec823c08ce1f9c0 WatchSource:0}: Error finding container 0efed9dd9d945d2f29991ccf98dcf428400a86faa2023a66bec823c08ce1f9c0: Status 404 returned error can't find the container with id 0efed9dd9d945d2f29991ccf98dcf428400a86faa2023a66bec823c08ce1f9c0
Nov 02 22:18:01.463578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.463551    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"]
Nov 02 22:18:01.577360 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:18:01.577253    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d57b7c2_237a_4e9f_9d2f_f5aa208b42a8.slice/crio-5b9709f3869f230a8e8dfb9d5101a35523b1194f546438f708b929fbea8aaad4 WatchSource:0}: Error finding container 5b9709f3869f230a8e8dfb9d5101a35523b1194f546438f708b929fbea8aaad4: Status 404 returned error can't find the container with id 5b9709f3869f230a8e8dfb9d5101a35523b1194f546438f708b929fbea8aaad4
Nov 02 22:18:01.579629 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.579607    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"]
Nov 02 22:18:01.604930 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.604898    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l" event={"ID":"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd","Type":"ContainerStarted","Data":"63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b"}
Nov 02 22:18:01.605068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.604959    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l" event={"ID":"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd","Type":"ContainerStarted","Data":"0efed9dd9d945d2f29991ccf98dcf428400a86faa2023a66bec823c08ce1f9c0"}
Nov 02 22:18:01.605164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.605132    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:01.606420 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.606397    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5" event={"ID":"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8","Type":"ContainerStarted","Data":"5b9709f3869f230a8e8dfb9d5101a35523b1194f546438f708b929fbea8aaad4"}
Nov 02 22:18:01.721467 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:01.721370    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l" podStartSLOduration=1.721355113 podStartE2EDuration="1.721355113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:01.717596214 +0000 UTC m=+4168.981132533" watchObservedRunningTime="2026-11-02 22:18:01.721355113 +0000 UTC m=+4168.984891431"
Nov 02 22:18:02.614848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:02.614811    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5" event={"ID":"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8","Type":"ContainerStarted","Data":"3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975"}
Nov 02 22:18:02.615307 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:02.614902    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:02.728562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:02.728511    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5" podStartSLOduration=2.728497937 podStartE2EDuration="2.728497937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:02.727267153 +0000 UTC m=+4169.990803472" watchObservedRunningTime="2026-11-02 22:18:02.728497937 +0000 UTC m=+4169.992034251"
Nov 02 22:18:12.618550 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:12.618521    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:18:13.624593 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:13.624562    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:18:14.113900 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:18:14.113857    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:18:27.436638 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:27.436601    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"]
Nov 02 22:18:27.437164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:27.436804    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy" containerID="cri-o://d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754" gracePeriod=30
Nov 02 22:18:32.709283 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.709219    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 22:18:32.786450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786416    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-credential-socket\") pod \"c4467661-9949-48a7-8d51-512c1abf63f4\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") "
Nov 02 22:18:32.786633 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786508    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-envoy\") pod \"c4467661-9949-48a7-8d51-512c1abf63f4\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") "
Nov 02 22:18:32.786633 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786613    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-data\") pod \"c4467661-9949-48a7-8d51-512c1abf63f4\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") "
Nov 02 22:18:32.786769 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786650    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-certs\") pod \"c4467661-9949-48a7-8d51-512c1abf63f4\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") "
Nov 02 22:18:32.786769 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786709    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-socket\") pod \"c4467661-9949-48a7-8d51-512c1abf63f4\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") "
Nov 02 22:18:32.786769 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786766    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4467661-9949-48a7-8d51-512c1abf63f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4467661-9949-48a7-8d51-512c1abf63f4-istio-podinfo\") pod \"c4467661-9949-48a7-8d51-512c1abf63f4\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") "
Nov 02 22:18:32.786943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786785    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-credential-socket" pod "c4467661-9949-48a7-8d51-512c1abf63f4" (UID: "c4467661-9949-48a7-8d51-512c1abf63f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:32.786943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786822    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-kube-api-access-8btfb\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-kube-api-access-8btfb\") pod \"c4467661-9949-48a7-8d51-512c1abf63f4\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") "
Nov 02 22:18:32.786943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786858    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4467661-9949-48a7-8d51-512c1abf63f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4467661-9949-48a7-8d51-512c1abf63f4-istiod-ca-cert\") pod \"c4467661-9949-48a7-8d51-512c1abf63f4\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") "
Nov 02 22:18:32.786943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786900    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-istio-token\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-istio-token\") pod \"c4467661-9949-48a7-8d51-512c1abf63f4\" (UID: \"c4467661-9949-48a7-8d51-512c1abf63f4\") "
Nov 02 22:18:32.786943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786915    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-certs" pod "c4467661-9949-48a7-8d51-512c1abf63f4" (UID: "c4467661-9949-48a7-8d51-512c1abf63f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:32.786943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786927    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-data" pod "c4467661-9949-48a7-8d51-512c1abf63f4" (UID: "c4467661-9949-48a7-8d51-512c1abf63f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:32.787290 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.786962    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-socket" pod "c4467661-9949-48a7-8d51-512c1abf63f4" (UID: "c4467661-9949-48a7-8d51-512c1abf63f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:32.787290 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.787272    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4467661-9949-48a7-8d51-512c1abf63f4-istiod-ca-cert" pod "c4467661-9949-48a7-8d51-512c1abf63f4" (UID: "c4467661-9949-48a7-8d51-512c1abf63f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:18:32.787527 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.787506    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4467661-9949-48a7-8d51-512c1abf63f4-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:32.787636 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.787530    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:32.787636 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.787545    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:32.787636 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.787557    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:32.787636 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.787572    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:32.789211 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.789121    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-istio-token" pod "c4467661-9949-48a7-8d51-512c1abf63f4" (UID: "c4467661-9949-48a7-8d51-512c1abf63f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:32.789345 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.789212    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4467661-9949-48a7-8d51-512c1abf63f4-istio-podinfo" pod "c4467661-9949-48a7-8d51-512c1abf63f4" (UID: "c4467661-9949-48a7-8d51-512c1abf63f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:18:32.789345 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.789227    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-envoy" pod "c4467661-9949-48a7-8d51-512c1abf63f4" (UID: "c4467661-9949-48a7-8d51-512c1abf63f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:18:32.789471 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.789339    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-kube-api-access-8btfb" pod "c4467661-9949-48a7-8d51-512c1abf63f4" (UID: "c4467661-9949-48a7-8d51-512c1abf63f4"). InnerVolumeSpecName "kube-api-access-8btfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:18:32.869327 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.869294    4528 generic.go:356] "Generic (PLEG): container finished" podID="c4467661-9949-48a7-8d51-512c1abf63f4" containerID="d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754" exitCode=0
Nov 02 22:18:32.869505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.869342    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"
Nov 02 22:18:32.869505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.869354    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" event={"ID":"c4467661-9949-48a7-8d51-512c1abf63f4","Type":"ContainerDied","Data":"d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754"}
Nov 02 22:18:32.869505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.869416    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5" event={"ID":"c4467661-9949-48a7-8d51-512c1abf63f4","Type":"ContainerDied","Data":"b178be0b816453ab4a716fa94e9b0031779036964f6f217afbfa1198cee3635f"}
Nov 02 22:18:32.869505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.869444    4528 scope.go:122] "RemoveContainer" containerID="d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754"
Nov 02 22:18:32.883406 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.883385    4528 scope.go:122] "RemoveContainer" containerID="d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754"
Nov 02 22:18:32.883744 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:18:32.883719    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754\": container with ID starting with d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754 not found: ID does not exist" containerID="d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754"
Nov 02 22:18:32.883854 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.883750    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754"} err="failed to get container status \"d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754\": rpc error: code = NotFound desc = could not find container \"d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754\": container with ID starting with d6a4c18db4e03ed74dbce28335520e3399ae53fa7b0b00762ee56dd41fdcd754 not found: ID does not exist"
Nov 02 22:18:32.888825 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.888790    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8btfb\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-kube-api-access-8btfb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:32.888966 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.888828    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4467661-9949-48a7-8d51-512c1abf63f4-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:32.888966 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.888843    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4467661-9949-48a7-8d51-512c1abf63f4-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:32.888966 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:32.888856    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4467661-9949-48a7-8d51-512c1abf63f4-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:18:33.045604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:33.045560    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"]
Nov 02 22:18:33.048502 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:33.048474    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-2wxp5"]
Nov 02 22:18:33.381455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:33.381372    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:18:34.914468 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:18:34.914433    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" path="/var/lib/kubelet/pods/c4467661-9949-48a7-8d51-512c1abf63f4/volumes"
Nov 02 22:18:44.115001 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:18:44.114962    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:19:14.114376 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:19:14.114337    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:19:21.571459 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.571416    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"]
Nov 02 22:19:21.572182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.572163    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy"
Nov 02 22:19:21.572182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.572182    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy"
Nov 02 22:19:21.572311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.572302    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4467661-9949-48a7-8d51-512c1abf63f4" containerName="istio-proxy"
Nov 02 22:19:21.575687 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.575667    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.626826 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.626792    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"]
Nov 02 22:19:21.672887 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.672853    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b58dcf14-knnzn\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.673084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.673064    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtsl\" (UniqueName: \"kubernetes.io/projected/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-kube-api-access-gmtsl\") pod \"maas-api-revoke-sub-b58dcf14-knnzn\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.673175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.673103    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b58dcf14-knnzn\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.673906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.673885    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:19:21.674006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.673991    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:19:21.679075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.679053    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:19:21.679174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.679081    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:19:21.774159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.774090    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b58dcf14-knnzn\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.774391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.774260    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtsl\" (UniqueName: \"kubernetes.io/projected/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-kube-api-access-gmtsl\") pod \"maas-api-revoke-sub-b58dcf14-knnzn\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.774391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.774292    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b58dcf14-knnzn\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.774905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.774882    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b58dcf14-knnzn\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.776590 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.776567    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b58dcf14-knnzn\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.831654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.831579    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtsl\" (UniqueName: \"kubernetes.io/projected/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-kube-api-access-gmtsl\") pod \"maas-api-revoke-sub-b58dcf14-knnzn\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:21.889580 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:21.889549    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:22.261776 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:19:22.261745    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578b01ef_ccca_4a0a_98be_dd05c5dc3a98.slice/crio-d5c032e6ad6bcafb3129c469af48ec95daf523e60dace6e1246b955f79f21786 WatchSource:0}: Error finding container d5c032e6ad6bcafb3129c469af48ec95daf523e60dace6e1246b955f79f21786: Status 404 returned error can't find the container with id d5c032e6ad6bcafb3129c469af48ec95daf523e60dace6e1246b955f79f21786
Nov 02 22:19:22.266600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:22.266569    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"]
Nov 02 22:19:23.224849 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:23.224819    4528 generic.go:356] "Generic (PLEG): container finished" podID="578b01ef-ccca-4a0a-98be-dd05c5dc3a98" containerID="3ecbe9a257244ad9ef4c19e266785ad848094040f0374841073251332a33e6cb" exitCode=0
Nov 02 22:19:23.225281 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:23.224897    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn" event={"ID":"578b01ef-ccca-4a0a-98be-dd05c5dc3a98","Type":"ContainerDied","Data":"3ecbe9a257244ad9ef4c19e266785ad848094040f0374841073251332a33e6cb"}
Nov 02 22:19:23.225281 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:23.224954    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn" event={"ID":"578b01ef-ccca-4a0a-98be-dd05c5dc3a98","Type":"ContainerStarted","Data":"d5c032e6ad6bcafb3129c469af48ec95daf523e60dace6e1246b955f79f21786"}
Nov 02 22:19:24.444899 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.444875    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:24.514811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.514784    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-xks-ca\") pod \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") "
Nov 02 22:19:24.515005 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.514843    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-kube-api-access-gmtsl\" (UniqueName: \"kubernetes.io/projected/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-kube-api-access-gmtsl\") pod \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") "
Nov 02 22:19:24.515005 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.514920    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-openshift-ca\") pod \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\" (UID: \"578b01ef-ccca-4a0a-98be-dd05c5dc3a98\") "
Nov 02 22:19:24.515368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.515343    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-openshift-ca" pod "578b01ef-ccca-4a0a-98be-dd05c5dc3a98" (UID: "578b01ef-ccca-4a0a-98be-dd05c5dc3a98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:19:24.516988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.516967    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-xks-ca" pod "578b01ef-ccca-4a0a-98be-dd05c5dc3a98" (UID: "578b01ef-ccca-4a0a-98be-dd05c5dc3a98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:19:24.517061 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.517018    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-kube-api-access-gmtsl" pod "578b01ef-ccca-4a0a-98be-dd05c5dc3a98" (UID: "578b01ef-ccca-4a0a-98be-dd05c5dc3a98"). InnerVolumeSpecName "kube-api-access-gmtsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:19:24.616264 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.616233    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:24.616264 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.616264    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:24.616477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:24.616273    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmtsl\" (UniqueName: \"kubernetes.io/projected/578b01ef-ccca-4a0a-98be-dd05c5dc3a98-kube-api-access-gmtsl\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:19:25.034405 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:25.034365    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:19:25.238710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:25.238636    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"
Nov 02 22:19:25.238710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:25.238646    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn" event={"ID":"578b01ef-ccca-4a0a-98be-dd05c5dc3a98","Type":"ContainerDied","Data":"d5c032e6ad6bcafb3129c469af48ec95daf523e60dace6e1246b955f79f21786"}
Nov 02 22:19:25.238710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:25.238687    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5c032e6ad6bcafb3129c469af48ec95daf523e60dace6e1246b955f79f21786"
Nov 02 22:19:36.371154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:36.371085    4528 scope.go:122] "RemoveContainer" containerID="7f18d0384b8f9c73535474e1c4d25a8cb3e37d9c65aa5e9620665b3c7920f5cd"
Nov 02 22:19:44.114248 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:19:44.114211    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:19:52.613716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.613676    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"]
Nov 02 22:19:52.614666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.614648    4528 cpu_manager.go:414] "containerMap: removing container" podUID="578b01ef-ccca-4a0a-98be-dd05c5dc3a98" containerName="revoke-keys"
Nov 02 22:19:52.614735 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.614668    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578b01ef-ccca-4a0a-98be-dd05c5dc3a98" containerName="revoke-keys"
Nov 02 22:19:52.614782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.614777    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="578b01ef-ccca-4a0a-98be-dd05c5dc3a98" containerName="revoke-keys"
Nov 02 22:19:52.618378 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.618355    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.669671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.669640    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"]
Nov 02 22:19:52.671412 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.671385    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6e535d\"/\"openshift-service-ca.crt\""
Nov 02 22:19:52.687045 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.687011    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-dshm\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.687405 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.687076    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tmp-dir\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.687405 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.687157    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2zt\" (UniqueName: \"kubernetes.io/projected/d2bee1ea-ab9d-421f-badf-59117f3de3ac-kube-api-access-fp2zt\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.687405 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.687225    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tls-certs\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.687405 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.687250    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-model-cache\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.687405 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.687288    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-home\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.715652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.715621    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6e535d\"/\"kube-root-ca.crt\""
Nov 02 22:19:52.721704 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.721678    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6e535d\"/\"default-dockercfg-wf5bh\""
Nov 02 22:19:52.721704 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.721685    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6e535d\"/\"rate-test-model-6e535d-kserve-self-signed-certs\""
Nov 02 22:19:52.793132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.791263    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-dshm\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.793132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.791358    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tmp-dir\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.793132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.791425    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2zt\" (UniqueName: \"kubernetes.io/projected/d2bee1ea-ab9d-421f-badf-59117f3de3ac-kube-api-access-fp2zt\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.793132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.791530    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tls-certs\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.793132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.791573    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-model-cache\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.793132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.791637    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-home\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.793132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.792147    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-home\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.793737 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.793710    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tmp-dir\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.812489 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.795948    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-model-cache\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.817842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.817808    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-dshm\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.822544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.822515    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tls-certs\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.849936 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.849903    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2zt\" (UniqueName: \"kubernetes.io/projected/d2bee1ea-ab9d-421f-badf-59117f3de3ac-kube-api-access-fp2zt\") pod \"rate-test-model-6e535d-kserve-6759d75857-fgbvv\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:52.932660 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:52.932556    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:19:53.319436 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:19:53.319401    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2bee1ea_ab9d_421f_badf_59117f3de3ac.slice/crio-51b3fd9c9aef8e95bc75b4f2e5a774ca07ddae26e5654ce4a8c3f4c070f7683d WatchSource:0}: Error finding container 51b3fd9c9aef8e95bc75b4f2e5a774ca07ddae26e5654ce4a8c3f4c070f7683d: Status 404 returned error can't find the container with id 51b3fd9c9aef8e95bc75b4f2e5a774ca07ddae26e5654ce4a8c3f4c070f7683d
Nov 02 22:19:53.320913 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:53.320886    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"]
Nov 02 22:19:53.436701 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:53.436663    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv" event={"ID":"d2bee1ea-ab9d-421f-badf-59117f3de3ac","Type":"ContainerStarted","Data":"0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91"}
Nov 02 22:19:53.436918 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:53.436715    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv" event={"ID":"d2bee1ea-ab9d-421f-badf-59117f3de3ac","Type":"ContainerStarted","Data":"51b3fd9c9aef8e95bc75b4f2e5a774ca07ddae26e5654ce4a8c3f4c070f7683d"}
Nov 02 22:19:53.552075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:53.552018    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv" podStartSLOduration=1.552003735 podStartE2EDuration="1.552003735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:53.55044737 +0000 UTC m=+4280.813983678" watchObservedRunningTime="2026-11-02 22:19:53.552003735 +0000 UTC m=+4280.815540051"
Nov 02 22:19:53.933217 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:19:53.933181    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:20:02.932741 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:02.932706    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:20:02.945446 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:02.945412    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:20:03.514878 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:03.514851    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:20:06.645377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:06.645337    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:20:14.113802 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:20:14.113764    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:20:44.115413 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:20:44.115374    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:20:56.768140 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.768094    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"]
Nov 02 22:20:56.772363 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.772343    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.794195 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.794160    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-data\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.794346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.794206    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b183f95a-965b-4052-b80f-f7fb5604a50c-istiod-ca-cert\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.794393 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.794353    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strbv\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-kube-api-access-strbv\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.794393 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.794379    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-credential-socket\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.794461 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.794409    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-token\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.794604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.794578    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-certs\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.794646 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.794627    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-envoy\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.794700 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.794681    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-podinfo\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.794771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.794753    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-socket\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.831538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.831498    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"]
Nov 02 22:20:56.874357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.874325    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3edc8c-openshift-default-dockercfg-jqchk\""
Nov 02 22:20:56.896455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.896407    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-data\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.896455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.896453    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b183f95a-965b-4052-b80f-f7fb5604a50c-istiod-ca-cert\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.898052 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.898024    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b183f95a-965b-4052-b80f-f7fb5604a50c-istiod-ca-cert\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.898546 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.898521    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-data\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.896557    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strbv\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-kube-api-access-strbv\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.955281    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-credential-socket\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.955320    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-token\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.955417    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-certs\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.955451    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-envoy\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.955481    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-podinfo\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.955579    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-socket\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.955967    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-socket\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.956221    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-credential-socket\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.959132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.956743    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-certs\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:56.985984 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.978202    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-envoy\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:57.004133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:56.994950    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-podinfo\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:57.028181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:57.018683    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-strbv\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-kube-api-access-strbv\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:57.042724 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:57.042688    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-token\") pod \"e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") " pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:57.086164 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:57.086128    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:57.563498 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:20:57.563460    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb183f95a_965b_4052_b80f_f7fb5604a50c.slice/crio-8542fb66eb89ec825c0d7ed5ecc3f1939b19048f54f45839710dc23b5a2553a7 WatchSource:0}: Error finding container 8542fb66eb89ec825c0d7ed5ecc3f1939b19048f54f45839710dc23b5a2553a7: Status 404 returned error can't find the container with id 8542fb66eb89ec825c0d7ed5ecc3f1939b19048f54f45839710dc23b5a2553a7
Nov 02 22:20:57.566842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:57.566808    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"]
Nov 02 22:20:57.566959 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:57.566922    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:20:57.567009 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:57.566983    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:20:57.567046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:57.567010    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:20:57.894917 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:57.894827    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" event={"ID":"b183f95a-965b-4052-b80f-f7fb5604a50c","Type":"ContainerStarted","Data":"07a034111a82ee829065e2f29be953689c3d34ffd4323f60967914000a71ac2d"}
Nov 02 22:20:57.894917 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:57.894882    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" event={"ID":"b183f95a-965b-4052-b80f-f7fb5604a50c","Type":"ContainerStarted","Data":"8542fb66eb89ec825c0d7ed5ecc3f1939b19048f54f45839710dc23b5a2553a7"}
Nov 02 22:20:58.019302 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:58.019245    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" podStartSLOduration=2.019229818 podStartE2EDuration="2.019229818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:58.015379476 +0000 UTC m=+4345.278915793" watchObservedRunningTime="2026-11-02 22:20:58.019229818 +0000 UTC m=+4345.282766135"
Nov 02 22:20:58.086779 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:58.086743    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:58.095903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:58.095871    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:58.903919 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:58.903887    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:20:58.907212 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:20:58.905042    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:21:03.396350 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.396318    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"]
Nov 02 22:21:03.400752 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.400732    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.604577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.601516    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b5e8794c-openshift-default-dockercfg-t5bh8\""
Nov 02 22:21:03.609390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.609355    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"]
Nov 02 22:21:03.626133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.611507    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-credential-socket\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.626133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.611573    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istiod-ca-cert\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.626133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.611627    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-envoy\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.626133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.611658    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-podinfo\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.626133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.611697    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-certs\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.626133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.611745    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-socket\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.626133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.611853    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn559\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-kube-api-access-pn559\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.626133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.611962    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-data\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.626133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.612008    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-token\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713200 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713099    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-certs\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713200 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713159    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-socket\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713458 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713229    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn559\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-kube-api-access-pn559\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713458 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713287    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-data\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713458 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713326    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-token\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713458 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713374    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-credential-socket\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713458 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713418    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istiod-ca-cert\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713751 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713538    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-envoy\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713751 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713547    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-socket\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713751 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713570    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-podinfo\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713917 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713773    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-data\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.713917 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713816    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-credential-socket\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.714050 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.713920    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-certs\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.714181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.714160    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istiod-ca-cert\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.716357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.716333    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-envoy\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.716504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.716487    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-podinfo\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.770970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.770931    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-token\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:03.771162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:03.771136    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn559\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-kube-api-access-pn559\") pod \"e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") " pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:04.015688 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.015657    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:04.457032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.456955    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"]
Nov 02 22:21:04.462258 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.462236    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:04.495821 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:21:04.495794    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ce6aaa_a760_4e75_963b_9c8cfc41fc2a.slice/crio-c595d07d1471af19b38c2ee34bbcd56d8a96954026649fa162459d057c5e9dac WatchSource:0}: Error finding container c595d07d1471af19b38c2ee34bbcd56d8a96954026649fa162459d057c5e9dac: Status 404 returned error can't find the container with id c595d07d1471af19b38c2ee34bbcd56d8a96954026649fa162459d057c5e9dac
Nov 02 22:21:04.498561 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.498538    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"]
Nov 02 22:21:04.498691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.498596    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:21:04.498691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.498675    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:21:04.498817 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.498717    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:21:04.513581 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.513558    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"]
Nov 02 22:21:04.514692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.514669    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3edc8c\""
Nov 02 22:21:04.514844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.514831    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3edc8c-dockercfg-wsptg\""
Nov 02 22:21:04.651170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.648734    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"]
Nov 02 22:21:04.664418 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.656985    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/f7d4c67d-1017-49a1-add1-0c40174a3427-kube-api-access-g2s7d\") pod \"payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs\" (UID: \"f7d4c67d-1017-49a1-add1-0c40174a3427\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:04.664418 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.657306    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d4c67d-1017-49a1-add1-0c40174a3427-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs\" (UID: \"f7d4c67d-1017-49a1-add1-0c40174a3427\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:04.664418 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.658906    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:04.689238 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.689210    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"]
Nov 02 22:21:04.761211 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.761180    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnj54\" (UniqueName: \"kubernetes.io/projected/25830df0-4c4b-4d46-b821-f050dff8bd51-kube-api-access-gnj54\") pod \"payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8\" (UID: \"25830df0-4c4b-4d46-b821-f050dff8bd51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:04.761394 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.761375    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d4c67d-1017-49a1-add1-0c40174a3427-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs\" (UID: \"f7d4c67d-1017-49a1-add1-0c40174a3427\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:04.761445 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.761427    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25830df0-4c4b-4d46-b821-f050dff8bd51-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8\" (UID: \"25830df0-4c4b-4d46-b821-f050dff8bd51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:04.761556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.761533    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/f7d4c67d-1017-49a1-add1-0c40174a3427-kube-api-access-g2s7d\") pod \"payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs\" (UID: \"f7d4c67d-1017-49a1-add1-0c40174a3427\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:04.762121 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.762078    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d4c67d-1017-49a1-add1-0c40174a3427-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs\" (UID: \"f7d4c67d-1017-49a1-add1-0c40174a3427\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:04.818630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.818597    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/f7d4c67d-1017-49a1-add1-0c40174a3427-kube-api-access-g2s7d\") pod \"payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs\" (UID: \"f7d4c67d-1017-49a1-add1-0c40174a3427\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:04.863325 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.863292    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnj54\" (UniqueName: \"kubernetes.io/projected/25830df0-4c4b-4d46-b821-f050dff8bd51-kube-api-access-gnj54\") pod \"payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8\" (UID: \"25830df0-4c4b-4d46-b821-f050dff8bd51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:04.863487 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.863396    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25830df0-4c4b-4d46-b821-f050dff8bd51-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8\" (UID: \"25830df0-4c4b-4d46-b821-f050dff8bd51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:04.864448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.864422    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25830df0-4c4b-4d46-b821-f050dff8bd51-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8\" (UID: \"25830df0-4c4b-4d46-b821-f050dff8bd51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:04.920365 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.920334    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnj54\" (UniqueName: \"kubernetes.io/projected/25830df0-4c4b-4d46-b821-f050dff8bd51-kube-api-access-gnj54\") pod \"payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8\" (UID: \"25830df0-4c4b-4d46-b821-f050dff8bd51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:04.943348 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.943316    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" event={"ID":"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a","Type":"ContainerStarted","Data":"0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139"}
Nov 02 22:21:04.943541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.943371    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" event={"ID":"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a","Type":"ContainerStarted","Data":"c595d07d1471af19b38c2ee34bbcd56d8a96954026649fa162459d057c5e9dac"}
Nov 02 22:21:04.990377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:04.990344    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:05.016372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.016280    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:05.018184 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.018155    4528 patch_prober.go:29] interesting pod/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body=
Nov 02 22:21:05.018328 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.018211    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused"
Nov 02 22:21:05.061542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.061484    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" podStartSLOduration=2.061462194 podStartE2EDuration="2.061462194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:05.060414628 +0000 UTC m=+4352.323950945" watchObservedRunningTime="2026-11-02 22:21:05.061462194 +0000 UTC m=+4352.324998512"
Nov 02 22:21:05.076496 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.076456    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:05.375027 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:21:05.374990    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25830df0_4c4b_4d46_b821_f050dff8bd51.slice/crio-456c8d90251062dbe9c9a6ef20364aae25c56eb92fd7e33ceda684b9c4e4bd05 WatchSource:0}: Error finding container 456c8d90251062dbe9c9a6ef20364aae25c56eb92fd7e33ceda684b9c4e4bd05: Status 404 returned error can't find the container with id 456c8d90251062dbe9c9a6ef20364aae25c56eb92fd7e33ceda684b9c4e4bd05
Nov 02 22:21:05.378279 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.378253    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"]
Nov 02 22:21:05.471630 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:21:05.471596    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d4c67d_1017_49a1_add1_0c40174a3427.slice/crio-7382c47575b39b88a64a5a08b9fd8229c2cf426bdffdd9d378c95b55fd5e397a WatchSource:0}: Error finding container 7382c47575b39b88a64a5a08b9fd8229c2cf426bdffdd9d378c95b55fd5e397a: Status 404 returned error can't find the container with id 7382c47575b39b88a64a5a08b9fd8229c2cf426bdffdd9d378c95b55fd5e397a
Nov 02 22:21:05.474810 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.474789    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"]
Nov 02 22:21:05.951187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.951143    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs" event={"ID":"f7d4c67d-1017-49a1-add1-0c40174a3427","Type":"ContainerStarted","Data":"64a966c7a7a66d953695a61e01518e82d0c7238bd9d6287471c25f62a13487fb"}
Nov 02 22:21:05.951387 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.951326    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:05.951387 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.951366    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs" event={"ID":"f7d4c67d-1017-49a1-add1-0c40174a3427","Type":"ContainerStarted","Data":"7382c47575b39b88a64a5a08b9fd8229c2cf426bdffdd9d378c95b55fd5e397a"}
Nov 02 22:21:05.952799 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.952775    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8" event={"ID":"25830df0-4c4b-4d46-b821-f050dff8bd51","Type":"ContainerStarted","Data":"14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4"}
Nov 02 22:21:05.952918 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.952816    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8" event={"ID":"25830df0-4c4b-4d46-b821-f050dff8bd51","Type":"ContainerStarted","Data":"456c8d90251062dbe9c9a6ef20364aae25c56eb92fd7e33ceda684b9c4e4bd05"}
Nov 02 22:21:05.952974 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:05.952930    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:06.021428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:06.021398    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:06.076272 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:06.076219    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs" podStartSLOduration=2.076206438 podStartE2EDuration="2.076206438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:06.073224288 +0000 UTC m=+4353.336760606" watchObservedRunningTime="2026-11-02 22:21:06.076206438 +0000 UTC m=+4353.339742755"
Nov 02 22:21:06.313512 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:06.313454    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8" podStartSLOduration=2.313439468 podStartE2EDuration="2.313439468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:06.312581733 +0000 UTC m=+4353.576118073" watchObservedRunningTime="2026-11-02 22:21:06.313439468 +0000 UTC m=+4353.576975810"
Nov 02 22:21:06.959461 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:06.959423    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:06.960213 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:06.960197    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:21:11.486776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.486734    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"]
Nov 02 22:21:11.493078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.493053    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:11.513474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.513441    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"]
Nov 02 22:21:11.559504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.559462    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b5e8794c\""
Nov 02 22:21:11.559504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.559478    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b5e8794c-dockercfg-p5lb8\""
Nov 02 22:21:11.602825 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.602789    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mpg\" (UniqueName: \"kubernetes.io/projected/39da56da-584c-4641-bab0-95ccf05bd997-kube-api-access-g5mpg\") pod \"payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5\" (UID: \"39da56da-584c-4641-bab0-95ccf05bd997\") " pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:11.602983 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.602909    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39da56da-584c-4641-bab0-95ccf05bd997-plugins-config-volume\") pod \"payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5\" (UID: \"39da56da-584c-4641-bab0-95ccf05bd997\") " pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:11.703733 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.703698    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39da56da-584c-4641-bab0-95ccf05bd997-plugins-config-volume\") pod \"payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5\" (UID: \"39da56da-584c-4641-bab0-95ccf05bd997\") " pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:11.703922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.703860    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mpg\" (UniqueName: \"kubernetes.io/projected/39da56da-584c-4641-bab0-95ccf05bd997-kube-api-access-g5mpg\") pod \"payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5\" (UID: \"39da56da-584c-4641-bab0-95ccf05bd997\") " pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:11.704486 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.704466    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39da56da-584c-4641-bab0-95ccf05bd997-plugins-config-volume\") pod \"payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5\" (UID: \"39da56da-584c-4641-bab0-95ccf05bd997\") " pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:11.760671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.760608    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mpg\" (UniqueName: \"kubernetes.io/projected/39da56da-584c-4641-bab0-95ccf05bd997-kube-api-access-g5mpg\") pod \"payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5\" (UID: \"39da56da-584c-4641-bab0-95ccf05bd997\") " pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:11.807727 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:11.807692    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:12.180681 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:21:12.180652    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39da56da_584c_4641_bab0_95ccf05bd997.slice/crio-ceceae2976b3b6f0821b74fbc60579429be6d4e5583d6994b5d62a207eda9a31 WatchSource:0}: Error finding container ceceae2976b3b6f0821b74fbc60579429be6d4e5583d6994b5d62a207eda9a31: Status 404 returned error can't find the container with id ceceae2976b3b6f0821b74fbc60579429be6d4e5583d6994b5d62a207eda9a31
Nov 02 22:21:12.182484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:12.182460    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"]
Nov 02 22:21:13.072844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:13.072809    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5" event={"ID":"39da56da-584c-4641-bab0-95ccf05bd997","Type":"ContainerStarted","Data":"f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46"}
Nov 02 22:21:13.073448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:13.072864    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5" event={"ID":"39da56da-584c-4641-bab0-95ccf05bd997","Type":"ContainerStarted","Data":"ceceae2976b3b6f0821b74fbc60579429be6d4e5583d6994b5d62a207eda9a31"}
Nov 02 22:21:13.073448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:13.072902    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:13.188598 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:13.188545    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5" podStartSLOduration=2.188529754 podStartE2EDuration="2.188529754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:13.186452744 +0000 UTC m=+4360.449989060" watchObservedRunningTime="2026-11-02 22:21:13.188529754 +0000 UTC m=+4360.452066100"
Nov 02 22:21:14.113599 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:21:14.113556    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:21:16.962137 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:16.962086    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:21:16.962579 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:16.962191    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:21:18.449580 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:18.449544    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:21:24.083086 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:24.083057    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:21:40.051793 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:40.051750    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"]
Nov 02 22:21:40.102086 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:40.102044    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-599a7646-cjxg2"]
Nov 02 22:21:40.913852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:40.913819    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2740c5-28e5-4c87-8457-aff181457bb5" path="/var/lib/kubelet/pods/ca2740c5-28e5-4c87-8457-aff181457bb5/volumes"
Nov 02 22:21:44.113558 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:21:44.113525    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:21:56.964030 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:21:56.963990    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:22:14.112835 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:22:14.112798    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:22:43.048183 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:22:43.048149    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"]
Nov 02 22:22:43.097875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:22:43.097839    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89fa6019-85fch"]
Nov 02 22:22:44.117086 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:22:44.117048    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:22:44.915123 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:22:44.915079    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df" path="/var/lib/kubelet/pods/ad3d3f57-3ef8-4cfb-9aee-1ac1ab0ce6df/volumes"
Nov 02 22:23:00.667626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:00.667590    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:23:11.897683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:11.897649    4528 generic.go:356] "Generic (PLEG): container finished" podID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerID="07a034111a82ee829065e2f29be953689c3d34ffd4323f60967914000a71ac2d" exitCode=0
Nov 02 22:23:11.898139 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:11.897721    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" event={"ID":"b183f95a-965b-4052-b80f-f7fb5604a50c","Type":"ContainerDied","Data":"07a034111a82ee829065e2f29be953689c3d34ffd4323f60967914000a71ac2d"}
Nov 02 22:23:11.898228 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:11.898149    4528 scope.go:122] "RemoveContainer" containerID="07a034111a82ee829065e2f29be953689c3d34ffd4323f60967914000a71ac2d"
Nov 02 22:23:11.899763 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:11.899739    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:23:11.899824 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:11.899803    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:23:11.899861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:11.899834    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:23:12.930511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:12.930472    4528 generic.go:356] "Generic (PLEG): container finished" podID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerID="0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139" exitCode=0
Nov 02 22:23:12.933605 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:12.933573    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" event={"ID":"b183f95a-965b-4052-b80f-f7fb5604a50c","Type":"ContainerStarted","Data":"cae172c79e766a2f0651a8f02b1486583ddedc7db7b115e151ef6d9478425ff2"}
Nov 02 22:23:12.933755 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:12.933626    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" event={"ID":"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a","Type":"ContainerDied","Data":"0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139"}
Nov 02 22:23:12.934042 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:12.934024    4528 scope.go:122] "RemoveContainer" containerID="0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139"
Nov 02 22:23:12.935713 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:12.935674    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:23:12.935841 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:12.935757    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:23:12.935841 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:12.935803    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:23:13.086243 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:13.086212    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:23:13.087575 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:13.087545    4528 patch_prober.go:29] interesting pod/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 02 22:23:13.087714 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:13.087602    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 02 22:23:13.905262 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:13.905226    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:23:13.960093 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:13.960050    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" event={"ID":"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a","Type":"ContainerStarted","Data":"33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b"}
Nov 02 22:23:14.028135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:14.026527    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:23:14.028135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:14.026655    4528 patch_prober.go:29] interesting pod/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body=
Nov 02 22:23:14.028135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:14.026695    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused"
Nov 02 22:23:14.087322 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:14.087288    4528 patch_prober.go:29] interesting pod/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 02 22:23:14.087495 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:14.087350    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 02 22:23:14.160432 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:23:14.160336    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:23:15.030133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:15.022593    4528 patch_prober.go:29] interesting pod/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body=
Nov 02 22:23:15.030133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:15.022650    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused"
Nov 02 22:23:15.087435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:15.087405    4528 patch_prober.go:29] interesting pod/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 02 22:23:15.087606 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:15.087452    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 02 22:23:16.022133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:16.021488    4528 patch_prober.go:29] interesting pod/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body=
Nov 02 22:23:16.022133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:16.021548    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused"
Nov 02 22:23:16.087189 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:16.087152    4528 patch_prober.go:29] interesting pod/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 02 22:23:16.087593 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:16.087214    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 02 22:23:17.018509 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:17.018467    4528 patch_prober.go:29] interesting pod/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body=
Nov 02 22:23:17.018800 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:17.018774    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused"
Nov 02 22:23:17.086818 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:17.086788    4528 patch_prober.go:29] interesting pod/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 02 22:23:17.087043 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:17.087022    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 02 22:23:18.022133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:18.021632    4528 patch_prober.go:29] interesting pod/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body=
Nov 02 22:23:18.022133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:18.021692    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused"
Nov 02 22:23:18.086544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:18.086508    4528 patch_prober.go:29] interesting pod/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 02 22:23:18.086737 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:18.086566    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 02 22:23:19.020050 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:19.020008    4528 patch_prober.go:29] interesting pod/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body=
Nov 02 22:23:19.020275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:19.020078    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused"
Nov 02 22:23:19.091614 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:19.091581    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:23:19.092654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:19.092632    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:23:20.021413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:20.021381    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:23:20.021901 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:20.021882    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:23:20.022658 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:20.022642    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:23:36.427279 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:36.427249    4528 scope.go:122] "RemoveContainer" containerID="eb309e52f4880ed340c2f2a9b343cdc5fe5f98e3d0bbb0799a095d182e93c8da"
Nov 02 22:23:38.998483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:23:38.998453    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:23:44.113000 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:23:44.112958    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:24:11.944116 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:11.944071    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"]
Nov 02 22:24:12.049892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:12.049857    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"]
Nov 02 22:24:14.112954 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:24:14.112913    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:24:16.944707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:16.944648    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l" podUID="5d233018-a0a5-4914-8b5e-fd6a5bfef1fd" containerName="payload-processing" containerID="cri-o://63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b" gracePeriod=25
Nov 02 22:24:17.051204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.051158    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5" podUID="6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8" containerName="payload-pre-processing" containerID="cri-o://3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975" gracePeriod=25
Nov 02 22:24:17.229531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.229496    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:24:17.267398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.267370    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-plugins-config-volume\") pod \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\" (UID: \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\") "
Nov 02 22:24:17.267578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.267557    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-kube-api-access-9lq8q\" (UniqueName: \"kubernetes.io/projected/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-kube-api-access-9lq8q\") pod \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\" (UID: \"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd\") "
Nov 02 22:24:17.268191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.267816    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-plugins-config-volume" pod "5d233018-a0a5-4914-8b5e-fd6a5bfef1fd" (UID: "5d233018-a0a5-4914-8b5e-fd6a5bfef1fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:24:17.268394 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.268363    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:17.272272 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.272242    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-kube-api-access-9lq8q" pod "5d233018-a0a5-4914-8b5e-fd6a5bfef1fd" (UID: "5d233018-a0a5-4914-8b5e-fd6a5bfef1fd"). InnerVolumeSpecName "kube-api-access-9lq8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:24:17.335244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.335222    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:24:17.368782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.368755    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-plugins-config-volume\") pod \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\" (UID: \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\") "
Nov 02 22:24:17.368974 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.368798    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-kube-api-access-66jrx\" (UniqueName: \"kubernetes.io/projected/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-kube-api-access-66jrx\") pod \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\" (UID: \"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8\") "
Nov 02 22:24:17.369147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.369096    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-plugins-config-volume" pod "6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8" (UID: "6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:24:17.369463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.369442    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:17.369522 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.369469    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lq8q\" (UniqueName: \"kubernetes.io/projected/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd-kube-api-access-9lq8q\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:17.370979 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.370958    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-kube-api-access-66jrx" pod "6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8" (UID: "6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8"). InnerVolumeSpecName "kube-api-access-66jrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:24:17.388650 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.388618    4528 generic.go:356] "Generic (PLEG): container finished" podID="5d233018-a0a5-4914-8b5e-fd6a5bfef1fd" containerID="63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b" exitCode=0
Nov 02 22:24:17.388774 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.388684    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"
Nov 02 22:24:17.388774 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.388695    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l" event={"ID":"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd","Type":"ContainerDied","Data":"63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b"}
Nov 02 22:24:17.388903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.388780    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l" event={"ID":"5d233018-a0a5-4914-8b5e-fd6a5bfef1fd","Type":"ContainerDied","Data":"0efed9dd9d945d2f29991ccf98dcf428400a86faa2023a66bec823c08ce1f9c0"}
Nov 02 22:24:17.388903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.388834    4528 scope.go:122] "RemoveContainer" containerID="63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b"
Nov 02 22:24:17.390362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.390308    4528 generic.go:356] "Generic (PLEG): container finished" podID="6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8" containerID="3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975" exitCode=0
Nov 02 22:24:17.390572 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.390488    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5" event={"ID":"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8","Type":"ContainerDied","Data":"3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975"}
Nov 02 22:24:17.390572 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.390492    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"
Nov 02 22:24:17.390572 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.390532    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5" event={"ID":"6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8","Type":"ContainerDied","Data":"5b9709f3869f230a8e8dfb9d5101a35523b1194f546438f708b929fbea8aaad4"}
Nov 02 22:24:17.401924 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.401907    4528 scope.go:122] "RemoveContainer" containerID="63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b"
Nov 02 22:24:17.402291 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:24:17.402271    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b\": container with ID starting with 63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b not found: ID does not exist" containerID="63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b"
Nov 02 22:24:17.402385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.402299    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b"} err="failed to get container status \"63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b\": rpc error: code = NotFound desc = could not find container \"63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b\": container with ID starting with 63ccc292a98820d3e0274ede62fdd89ebdd9fc074cecaad930d48108934a0d1b not found: ID does not exist"
Nov 02 22:24:17.402385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.402313    4528 scope.go:122] "RemoveContainer" containerID="3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975"
Nov 02 22:24:17.411184 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.411169    4528 scope.go:122] "RemoveContainer" containerID="3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975"
Nov 02 22:24:17.411419 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:24:17.411398    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975\": container with ID starting with 3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975 not found: ID does not exist" containerID="3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975"
Nov 02 22:24:17.411474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.411426    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975"} err="failed to get container status \"3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975\": rpc error: code = NotFound desc = could not find container \"3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975\": container with ID starting with 3db2c22a1f1dd47f6a8545f45ebcd58ce1a8cf5662121e21823b582da26fc975 not found: ID does not exist"
Nov 02 22:24:17.455931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.455875    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"]
Nov 02 22:24:17.470613 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.470585    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66jrx\" (UniqueName: \"kubernetes.io/projected/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8-kube-api-access-66jrx\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:17.568694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.568663    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"]
Nov 02 22:24:17.574582 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.572770    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db69f7-69b9bf697f-br6f5"]
Nov 02 22:24:17.792248 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.792212    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"]
Nov 02 22:24:17.797844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:17.797818    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db69f7-7cdc6d6d6f-cll4l"]
Nov 02 22:24:18.915692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:18.915659    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d233018-a0a5-4914-8b5e-fd6a5bfef1fd" path="/var/lib/kubelet/pods/5d233018-a0a5-4914-8b5e-fd6a5bfef1fd/volumes"
Nov 02 22:24:18.916076 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:18.916052    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8" path="/var/lib/kubelet/pods/6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8/volumes"
Nov 02 22:24:22.456820 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:22.456777    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5" podUID="39da56da-584c-4641-bab0-95ccf05bd997" containerName="payload-processing" containerID="cri-o://f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46" gracePeriod=25
Nov 02 22:24:22.726961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:22.726935    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:24:22.768201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:22.768173    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39da56da-584c-4641-bab0-95ccf05bd997-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39da56da-584c-4641-bab0-95ccf05bd997-plugins-config-volume\") pod \"39da56da-584c-4641-bab0-95ccf05bd997\" (UID: \"39da56da-584c-4641-bab0-95ccf05bd997\") "
Nov 02 22:24:22.768347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:22.768276    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39da56da-584c-4641-bab0-95ccf05bd997-kube-api-access-g5mpg\" (UniqueName: \"kubernetes.io/projected/39da56da-584c-4641-bab0-95ccf05bd997-kube-api-access-g5mpg\") pod \"39da56da-584c-4641-bab0-95ccf05bd997\" (UID: \"39da56da-584c-4641-bab0-95ccf05bd997\") "
Nov 02 22:24:22.768606 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:22.768581    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39da56da-584c-4641-bab0-95ccf05bd997-plugins-config-volume" pod "39da56da-584c-4641-bab0-95ccf05bd997" (UID: "39da56da-584c-4641-bab0-95ccf05bd997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:24:22.770311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:22.770290    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39da56da-584c-4641-bab0-95ccf05bd997-kube-api-access-g5mpg" pod "39da56da-584c-4641-bab0-95ccf05bd997" (UID: "39da56da-584c-4641-bab0-95ccf05bd997"). InnerVolumeSpecName "kube-api-access-g5mpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:24:22.869811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:22.869771    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39da56da-584c-4641-bab0-95ccf05bd997-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:22.869811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:22.869801    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5mpg\" (UniqueName: \"kubernetes.io/projected/39da56da-584c-4641-bab0-95ccf05bd997-kube-api-access-g5mpg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:24:23.432869 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:23.432835    4528 generic.go:356] "Generic (PLEG): container finished" podID="39da56da-584c-4641-bab0-95ccf05bd997" containerID="f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46" exitCode=0
Nov 02 22:24:23.433058 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:23.432875    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"
Nov 02 22:24:23.433058 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:23.432908    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5" event={"ID":"39da56da-584c-4641-bab0-95ccf05bd997","Type":"ContainerDied","Data":"f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46"}
Nov 02 22:24:23.433058 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:23.432956    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5" event={"ID":"39da56da-584c-4641-bab0-95ccf05bd997","Type":"ContainerDied","Data":"ceceae2976b3b6f0821b74fbc60579429be6d4e5583d6994b5d62a207eda9a31"}
Nov 02 22:24:23.433058 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:23.432977    4528 scope.go:122] "RemoveContainer" containerID="f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46"
Nov 02 22:24:23.443129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:23.443090    4528 scope.go:122] "RemoveContainer" containerID="f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46"
Nov 02 22:24:23.443390 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:24:23.443369    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46\": container with ID starting with f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46 not found: ID does not exist" containerID="f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46"
Nov 02 22:24:23.443467 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:23.443398    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46"} err="failed to get container status \"f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46\": rpc error: code = NotFound desc = could not find container \"f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46\": container with ID starting with f7e99bd4856a85866065414f2cea1a0eaa96425c3553b85f157feaa45419ce46 not found: ID does not exist"
Nov 02 22:24:23.599196 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:23.599163    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"]
Nov 02 22:24:23.649154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:23.649104    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5e8794c-784dd98f49-blrs5"]
Nov 02 22:24:24.914059 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:24.914026    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39da56da-584c-4641-bab0-95ccf05bd997" path="/var/lib/kubelet/pods/39da56da-584c-4641-bab0-95ccf05bd997/volumes"
Nov 02 22:24:25.047127 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:25.047087    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"]
Nov 02 22:24:25.097365 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:25.097335    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b58dcf14-knnzn"]
Nov 02 22:24:26.913965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:26.913929    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578b01ef-ccca-4a0a-98be-dd05c5dc3a98" path="/var/lib/kubelet/pods/578b01ef-ccca-4a0a-98be-dd05c5dc3a98/volumes"
Nov 02 22:24:36.450738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:24:36.450708    4528 scope.go:122] "RemoveContainer" containerID="e1ae8c640321bc305ed37f8e2385d730201eb09a02632dd9aec7257735a86ba8"
Nov 02 22:24:44.112477 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:24:44.112432    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:25:14.113960 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:25:14.113917    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:25:24.899657 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:24.899623    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:25:33.545668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.545633    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"]
Nov 02 22:25:33.546243 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.546228    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8" containerName="payload-pre-processing"
Nov 02 22:25:33.546243 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.546243    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8" containerName="payload-pre-processing"
Nov 02 22:25:33.546339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.546255    4528 cpu_manager.go:414] "containerMap: removing container" podUID="39da56da-584c-4641-bab0-95ccf05bd997" containerName="payload-processing"
Nov 02 22:25:33.546339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.546260    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39da56da-584c-4641-bab0-95ccf05bd997" containerName="payload-processing"
Nov 02 22:25:33.546339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.546269    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5d233018-a0a5-4914-8b5e-fd6a5bfef1fd" containerName="payload-processing"
Nov 02 22:25:33.546339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.546274    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d233018-a0a5-4914-8b5e-fd6a5bfef1fd" containerName="payload-processing"
Nov 02 22:25:33.546470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.546372    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d57b7c2-237a-4e9f-9d2f-f5aa208b42a8" containerName="payload-pre-processing"
Nov 02 22:25:33.546470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.546379    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="39da56da-584c-4641-bab0-95ccf05bd997" containerName="payload-processing"
Nov 02 22:25:33.546470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.546388    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d233018-a0a5-4914-8b5e-fd6a5bfef1fd" containerName="payload-processing"
Nov 02 22:25:33.549986 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.549971    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.602554 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.602520    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"]
Nov 02 22:25:33.647854 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.647824    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:25:33.648006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.647858    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:25:33.648006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.647895    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:25:33.652418 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.652398    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:25:33.672169 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.672140    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58996c83-lh5bc\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.672346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.672329    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58996c83-lh5bc\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.672432 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.672359    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngvr\" (UniqueName: \"kubernetes.io/projected/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-kube-api-access-wngvr\") pod \"maas-api-revoke-sub-58996c83-lh5bc\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.773648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.773625    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58996c83-lh5bc\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.773782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.773721    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58996c83-lh5bc\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.773782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.773740    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngvr\" (UniqueName: \"kubernetes.io/projected/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-kube-api-access-wngvr\") pod \"maas-api-revoke-sub-58996c83-lh5bc\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.774267 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.774246    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58996c83-lh5bc\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.776076 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.776054    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58996c83-lh5bc\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.830282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.830216    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngvr\" (UniqueName: \"kubernetes.io/projected/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-kube-api-access-wngvr\") pod \"maas-api-revoke-sub-58996c83-lh5bc\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:33.871250 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:33.871228    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:34.241839 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:25:34.241801    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5495ee2c_12e1_43de_8d3c_1db522e7dc4f.slice/crio-45a6dc5e6cfb0cc8bcf67aa888f5a08f436091f6601b877b14f10f2366a92b17 WatchSource:0}: Error finding container 45a6dc5e6cfb0cc8bcf67aa888f5a08f436091f6601b877b14f10f2366a92b17: Status 404 returned error can't find the container with id 45a6dc5e6cfb0cc8bcf67aa888f5a08f436091f6601b877b14f10f2366a92b17
Nov 02 22:25:34.243665 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:34.243641    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"]
Nov 02 22:25:34.897388 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:34.897295    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc" event={"ID":"5495ee2c-12e1-43de-8d3c-1db522e7dc4f","Type":"ContainerStarted","Data":"a3420ab7db5e937b53ff27cc1bdfbc18f95ae04086b8a5a86e922e5514539841"}
Nov 02 22:25:34.897388 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:34.897346    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc" event={"ID":"5495ee2c-12e1-43de-8d3c-1db522e7dc4f","Type":"ContainerStarted","Data":"45a6dc5e6cfb0cc8bcf67aa888f5a08f436091f6601b877b14f10f2366a92b17"}
Nov 02 22:25:35.012904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:35.012855    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc" podStartSLOduration=2.012841536 podStartE2EDuration="2.012841536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:35.010476061 +0000 UTC m=+4622.274012379" watchObservedRunningTime="2026-11-02 22:25:35.012841536 +0000 UTC m=+4622.276377879"
Nov 02 22:25:35.905556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:35.905525    4528 generic.go:356] "Generic (PLEG): container finished" podID="5495ee2c-12e1-43de-8d3c-1db522e7dc4f" containerID="a3420ab7db5e937b53ff27cc1bdfbc18f95ae04086b8a5a86e922e5514539841" exitCode=0
Nov 02 22:25:35.905966 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:35.905612    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc" event={"ID":"5495ee2c-12e1-43de-8d3c-1db522e7dc4f","Type":"ContainerDied","Data":"a3420ab7db5e937b53ff27cc1bdfbc18f95ae04086b8a5a86e922e5514539841"}
Nov 02 22:25:36.488826 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:36.488793    4528 scope.go:122] "RemoveContainer" containerID="3ecbe9a257244ad9ef4c19e266785ad848094040f0374841073251332a33e6cb"
Nov 02 22:25:37.069461 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.069436    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:37.218650 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.218557    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-kube-api-access-wngvr\" (UniqueName: \"kubernetes.io/projected/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-kube-api-access-wngvr\") pod \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") "
Nov 02 22:25:37.218650 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.218638    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-openshift-ca\") pod \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") "
Nov 02 22:25:37.218897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.218773    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-xks-ca\") pod \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\" (UID: \"5495ee2c-12e1-43de-8d3c-1db522e7dc4f\") "
Nov 02 22:25:37.219022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.218997    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-openshift-ca" pod "5495ee2c-12e1-43de-8d3c-1db522e7dc4f" (UID: "5495ee2c-12e1-43de-8d3c-1db522e7dc4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:37.219373 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.219348    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:37.220738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.220711    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-xks-ca" pod "5495ee2c-12e1-43de-8d3c-1db522e7dc4f" (UID: "5495ee2c-12e1-43de-8d3c-1db522e7dc4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:37.220847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.220816    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-kube-api-access-wngvr" pod "5495ee2c-12e1-43de-8d3c-1db522e7dc4f" (UID: "5495ee2c-12e1-43de-8d3c-1db522e7dc4f"). InnerVolumeSpecName "kube-api-access-wngvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:37.320567 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.320543    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngvr\" (UniqueName: \"kubernetes.io/projected/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-kube-api-access-wngvr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:37.320567 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.320565    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5495ee2c-12e1-43de-8d3c-1db522e7dc4f-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:37.921808 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.921771    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc" event={"ID":"5495ee2c-12e1-43de-8d3c-1db522e7dc4f","Type":"ContainerDied","Data":"45a6dc5e6cfb0cc8bcf67aa888f5a08f436091f6601b877b14f10f2366a92b17"}
Nov 02 22:25:37.921988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.921811    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"
Nov 02 22:25:37.921988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:37.921848    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45a6dc5e6cfb0cc8bcf67aa888f5a08f436091f6601b877b14f10f2366a92b17"
Nov 02 22:25:39.583766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:39.583734    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"]
Nov 02 22:25:39.690094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:39.690068    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"]
Nov 02 22:25:42.623304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.623267    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"]
Nov 02 22:25:42.624638 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.624608    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5495ee2c-12e1-43de-8d3c-1db522e7dc4f" containerName="revoke-keys"
Nov 02 22:25:42.624638 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.624635    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5495ee2c-12e1-43de-8d3c-1db522e7dc4f" containerName="revoke-keys"
Nov 02 22:25:42.626449 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.624997    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5495ee2c-12e1-43de-8d3c-1db522e7dc4f" containerName="revoke-keys"
Nov 02 22:25:42.630102 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.630076    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:42.674566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.674531    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"]
Nov 02 22:25:42.682928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.682901    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:25:42.683075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.682993    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:25:42.721663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.721643    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:25:42.721770 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.721679    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:25:42.799939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.799911    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:42.800091 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.799977    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:42.800233 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.800211    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwrj\" (UniqueName: \"kubernetes.io/projected/6a7f7c74-dced-42c9-9240-a666235dec87-kube-api-access-mfwrj\") pod \"maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:42.900956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.900871    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwrj\" (UniqueName: \"kubernetes.io/projected/6a7f7c74-dced-42c9-9240-a666235dec87-kube-api-access-mfwrj\") pod \"maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:42.901141 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.900973    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:42.901141 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.901021    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:42.901689 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.901668    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:42.903294 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.903275    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:42.957772 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:42.957747    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwrj\" (UniqueName: \"kubernetes.io/projected/6a7f7c74-dced-42c9-9240-a666235dec87-kube-api-access-mfwrj\") pod \"maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:43.243496 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:43.243417    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:43.611931 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:25:43.611900    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7f7c74_dced_42c9_9240_a666235dec87.slice/crio-f74f3f0b94960af636374d203567e81940db9a974723a9869de83a81552a7897 WatchSource:0}: Error finding container f74f3f0b94960af636374d203567e81940db9a974723a9869de83a81552a7897: Status 404 returned error can't find the container with id f74f3f0b94960af636374d203567e81940db9a974723a9869de83a81552a7897
Nov 02 22:25:43.616643 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:43.616619    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"]
Nov 02 22:25:43.958302 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:43.958273    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k" event={"ID":"6a7f7c74-dced-42c9-9240-a666235dec87","Type":"ContainerStarted","Data":"e102eab0f02795782cedc00c48ec5878c9192bc403a4962aaad23b33ed1ac547"}
Nov 02 22:25:43.958625 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:43.958323    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k" event={"ID":"6a7f7c74-dced-42c9-9240-a666235dec87","Type":"ContainerStarted","Data":"f74f3f0b94960af636374d203567e81940db9a974723a9869de83a81552a7897"}
Nov 02 22:25:44.071439 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.071389    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k" podStartSLOduration=2.071375154 podStartE2EDuration="2.071375154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:44.070812442 +0000 UTC m=+4631.334348759" watchObservedRunningTime="2026-11-02 22:25:44.071375154 +0000 UTC m=+4631.334911471"
Nov 02 22:25:44.112671 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:25:44.112603    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:25:44.375392 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.375306    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"]
Nov 02 22:25:44.379600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.379577    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.418997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.418965    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfzt7\" (UniqueName: \"kubernetes.io/projected/0099729c-ec0f-4b6b-b375-0b743fd0e19c-kube-api-access-dfzt7\") pod \"maas-api-revoke-sub-1339d374-57kst\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.419196 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.419156    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1339d374-57kst\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.419276 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.419203    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1339d374-57kst\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.429537 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.429511    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"]
Nov 02 22:25:44.520491 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.520466    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1339d374-57kst\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.520643 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.520522    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1339d374-57kst\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.520709 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.520686    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfzt7\" (UniqueName: \"kubernetes.io/projected/0099729c-ec0f-4b6b-b375-0b743fd0e19c-kube-api-access-dfzt7\") pod \"maas-api-revoke-sub-1339d374-57kst\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.521172 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.521146    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1339d374-57kst\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.522668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.522647    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1339d374-57kst\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.577104 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.577078    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfzt7\" (UniqueName: \"kubernetes.io/projected/0099729c-ec0f-4b6b-b375-0b743fd0e19c-kube-api-access-dfzt7\") pod \"maas-api-revoke-sub-1339d374-57kst\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.584569 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.584546    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8" podUID="25830df0-4c4b-4d46-b821-f050dff8bd51" containerName="payload-processing" containerID="cri-o://14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4" gracePeriod=25
Nov 02 22:25:44.690559 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.690481    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs" podUID="f7d4c67d-1017-49a1-add1-0c40174a3427" containerName="payload-pre-processing" containerID="cri-o://64a966c7a7a66d953695a61e01518e82d0c7238bd9d6287471c25f62a13487fb" gracePeriod=25
Nov 02 22:25:44.694360 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.694143    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:44.878975 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.878947    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:25:44.926647 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.926620    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25830df0-4c4b-4d46-b821-f050dff8bd51-kube-api-access-gnj54\" (UniqueName: \"kubernetes.io/projected/25830df0-4c4b-4d46-b821-f050dff8bd51-kube-api-access-gnj54\") pod \"25830df0-4c4b-4d46-b821-f050dff8bd51\" (UID: \"25830df0-4c4b-4d46-b821-f050dff8bd51\") "
Nov 02 22:25:44.926840 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.926750    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25830df0-4c4b-4d46-b821-f050dff8bd51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25830df0-4c4b-4d46-b821-f050dff8bd51-plugins-config-volume\") pod \"25830df0-4c4b-4d46-b821-f050dff8bd51\" (UID: \"25830df0-4c4b-4d46-b821-f050dff8bd51\") "
Nov 02 22:25:44.927123 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.927091    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25830df0-4c4b-4d46-b821-f050dff8bd51-plugins-config-volume" pod "25830df0-4c4b-4d46-b821-f050dff8bd51" (UID: "25830df0-4c4b-4d46-b821-f050dff8bd51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:44.928788 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.928765    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25830df0-4c4b-4d46-b821-f050dff8bd51-kube-api-access-gnj54" pod "25830df0-4c4b-4d46-b821-f050dff8bd51" (UID: "25830df0-4c4b-4d46-b821-f050dff8bd51"). InnerVolumeSpecName "kube-api-access-gnj54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:44.965680 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.965654    4528 generic.go:356] "Generic (PLEG): container finished" podID="6a7f7c74-dced-42c9-9240-a666235dec87" containerID="e102eab0f02795782cedc00c48ec5878c9192bc403a4962aaad23b33ed1ac547" exitCode=0
Nov 02 22:25:44.966026 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.965703    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k" event={"ID":"6a7f7c74-dced-42c9-9240-a666235dec87","Type":"ContainerDied","Data":"e102eab0f02795782cedc00c48ec5878c9192bc403a4962aaad23b33ed1ac547"}
Nov 02 22:25:44.967177 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.967154    4528 generic.go:356] "Generic (PLEG): container finished" podID="25830df0-4c4b-4d46-b821-f050dff8bd51" containerID="14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4" exitCode=0
Nov 02 22:25:44.967292 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.967202    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"
Nov 02 22:25:44.967292 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.967237    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8" event={"ID":"25830df0-4c4b-4d46-b821-f050dff8bd51","Type":"ContainerDied","Data":"14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4"}
Nov 02 22:25:44.967427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.967311    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8" event={"ID":"25830df0-4c4b-4d46-b821-f050dff8bd51","Type":"ContainerDied","Data":"456c8d90251062dbe9c9a6ef20364aae25c56eb92fd7e33ceda684b9c4e4bd05"}
Nov 02 22:25:44.967427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.967364    4528 scope.go:122] "RemoveContainer" containerID="14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4"
Nov 02 22:25:44.968783 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.968765    4528 generic.go:356] "Generic (PLEG): container finished" podID="f7d4c67d-1017-49a1-add1-0c40174a3427" containerID="64a966c7a7a66d953695a61e01518e82d0c7238bd9d6287471c25f62a13487fb" exitCode=0
Nov 02 22:25:44.968865 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.968830    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs" event={"ID":"f7d4c67d-1017-49a1-add1-0c40174a3427","Type":"ContainerDied","Data":"64a966c7a7a66d953695a61e01518e82d0c7238bd9d6287471c25f62a13487fb"}
Nov 02 22:25:44.968939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.968877    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs" event={"ID":"f7d4c67d-1017-49a1-add1-0c40174a3427","Type":"ContainerDied","Data":"7382c47575b39b88a64a5a08b9fd8229c2cf426bdffdd9d378c95b55fd5e397a"}
Nov 02 22:25:44.968939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.968902    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7382c47575b39b88a64a5a08b9fd8229c2cf426bdffdd9d378c95b55fd5e397a"
Nov 02 22:25:44.987982 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.987967    4528 scope.go:122] "RemoveContainer" containerID="14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4"
Nov 02 22:25:44.988291 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:25:44.988263    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4\": container with ID starting with 14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4 not found: ID does not exist" containerID="14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4"
Nov 02 22:25:44.988384 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.988300    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4"} err="failed to get container status \"14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4\": rpc error: code = NotFound desc = could not find container \"14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4\": container with ID starting with 14fbf792982368afc7fcd97439a1bb96abf75d83015609d09db82b8355fd71e4 not found: ID does not exist"
Nov 02 22:25:44.993267 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:44.993248    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:25:45.028295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.028268    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d4c67d-1017-49a1-add1-0c40174a3427-kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/f7d4c67d-1017-49a1-add1-0c40174a3427-kube-api-access-g2s7d\") pod \"f7d4c67d-1017-49a1-add1-0c40174a3427\" (UID: \"f7d4c67d-1017-49a1-add1-0c40174a3427\") "
Nov 02 22:25:45.028407 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.028366    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d4c67d-1017-49a1-add1-0c40174a3427-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d4c67d-1017-49a1-add1-0c40174a3427-plugins-config-volume\") pod \"f7d4c67d-1017-49a1-add1-0c40174a3427\" (UID: \"f7d4c67d-1017-49a1-add1-0c40174a3427\") "
Nov 02 22:25:45.028805 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.028775    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d4c67d-1017-49a1-add1-0c40174a3427-plugins-config-volume" pod "f7d4c67d-1017-49a1-add1-0c40174a3427" (UID: "f7d4c67d-1017-49a1-add1-0c40174a3427"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:45.029419 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.029119    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnj54\" (UniqueName: \"kubernetes.io/projected/25830df0-4c4b-4d46-b821-f050dff8bd51-kube-api-access-gnj54\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:45.029419 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.029143    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d4c67d-1017-49a1-add1-0c40174a3427-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:45.029419 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.029169    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25830df0-4c4b-4d46-b821-f050dff8bd51-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:45.031127 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.031082    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d4c67d-1017-49a1-add1-0c40174a3427-kube-api-access-g2s7d" pod "f7d4c67d-1017-49a1-add1-0c40174a3427" (UID: "f7d4c67d-1017-49a1-add1-0c40174a3427"). InnerVolumeSpecName "kube-api-access-g2s7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:45.099044 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:25:45.098420    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0099729c_ec0f_4b6b_b375_0b743fd0e19c.slice/crio-1a1630c06f888542e9aebd8587db777d8df45e13101aae80557b2797f6d61a2a WatchSource:0}: Error finding container 1a1630c06f888542e9aebd8587db777d8df45e13101aae80557b2797f6d61a2a: Status 404 returned error can't find the container with id 1a1630c06f888542e9aebd8587db777d8df45e13101aae80557b2797f6d61a2a
Nov 02 22:25:45.100703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.100679    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"]
Nov 02 22:25:45.130382 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.130357    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/f7d4c67d-1017-49a1-add1-0c40174a3427-kube-api-access-g2s7d\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:45.333897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.333864    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"]
Nov 02 22:25:45.337574 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.337546    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3edc8c-5b49cf55cf-2tct8"]
Nov 02 22:25:45.977352 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.977306    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst" event={"ID":"0099729c-ec0f-4b6b-b375-0b743fd0e19c","Type":"ContainerStarted","Data":"730ed30632ea122c70a95d118f92f17ccb04686b31a690e15b6c98d87743e81f"}
Nov 02 22:25:45.977765 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.977363    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst" event={"ID":"0099729c-ec0f-4b6b-b375-0b743fd0e19c","Type":"ContainerStarted","Data":"1a1630c06f888542e9aebd8587db777d8df45e13101aae80557b2797f6d61a2a"}
Nov 02 22:25:45.978397 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:45.978375    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"
Nov 02 22:25:46.094014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.093962    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst" podStartSLOduration=2.093947912 podStartE2EDuration="2.093947912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:46.090167473 +0000 UTC m=+4633.353703791" watchObservedRunningTime="2026-11-02 22:25:46.093947912 +0000 UTC m=+4633.357484266"
Nov 02 22:25:46.144291 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.144270    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:46.204274 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.204245    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"]
Nov 02 22:25:46.207663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.207642    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3edc8c-78489f9f75-9xdjs"]
Nov 02 22:25:46.242470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.242419    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-xks-ca\") pod \"6a7f7c74-dced-42c9-9240-a666235dec87\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") "
Nov 02 22:25:46.242470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.242451    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7f7c74-dced-42c9-9240-a666235dec87-kube-api-access-mfwrj\" (UniqueName: \"kubernetes.io/projected/6a7f7c74-dced-42c9-9240-a666235dec87-kube-api-access-mfwrj\") pod \"6a7f7c74-dced-42c9-9240-a666235dec87\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") "
Nov 02 22:25:46.242619 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.242551    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-openshift-ca\") pod \"6a7f7c74-dced-42c9-9240-a666235dec87\" (UID: \"6a7f7c74-dced-42c9-9240-a666235dec87\") "
Nov 02 22:25:46.242896 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.242873    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-openshift-ca" pod "6a7f7c74-dced-42c9-9240-a666235dec87" (UID: "6a7f7c74-dced-42c9-9240-a666235dec87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:46.243139 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.243103    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:46.244409 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.244386    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-xks-ca" pod "6a7f7c74-dced-42c9-9240-a666235dec87" (UID: "6a7f7c74-dced-42c9-9240-a666235dec87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:46.244621 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.244605    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7f7c74-dced-42c9-9240-a666235dec87-kube-api-access-mfwrj" pod "6a7f7c74-dced-42c9-9240-a666235dec87" (UID: "6a7f7c74-dced-42c9-9240-a666235dec87"). InnerVolumeSpecName "kube-api-access-mfwrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:46.344921 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.344878    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7f7c74-dced-42c9-9240-a666235dec87-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:46.345073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.344924    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwrj\" (UniqueName: \"kubernetes.io/projected/6a7f7c74-dced-42c9-9240-a666235dec87-kube-api-access-mfwrj\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:46.916061 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.916028    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25830df0-4c4b-4d46-b821-f050dff8bd51" path="/var/lib/kubelet/pods/25830df0-4c4b-4d46-b821-f050dff8bd51/volumes"
Nov 02 22:25:46.916495 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.916479    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d4c67d-1017-49a1-add1-0c40174a3427" path="/var/lib/kubelet/pods/f7d4c67d-1017-49a1-add1-0c40174a3427/volumes"
Nov 02 22:25:46.985934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.985901    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"
Nov 02 22:25:46.986330 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.985948    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k" event={"ID":"6a7f7c74-dced-42c9-9240-a666235dec87","Type":"ContainerDied","Data":"f74f3f0b94960af636374d203567e81940db9a974723a9869de83a81552a7897"}
Nov 02 22:25:46.986330 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.985998    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f74f3f0b94960af636374d203567e81940db9a974723a9869de83a81552a7897"
Nov 02 22:25:46.987691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.987666    4528 generic.go:356] "Generic (PLEG): container finished" podID="0099729c-ec0f-4b6b-b375-0b743fd0e19c" containerID="730ed30632ea122c70a95d118f92f17ccb04686b31a690e15b6c98d87743e81f" exitCode=0
Nov 02 22:25:46.987856 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:46.987836    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst" event={"ID":"0099729c-ec0f-4b6b-b375-0b743fd0e19c","Type":"ContainerDied","Data":"730ed30632ea122c70a95d118f92f17ccb04686b31a690e15b6c98d87743e81f"}
Nov 02 22:25:48.148649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.148624    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:48.164332 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.164310    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0099729c-ec0f-4b6b-b375-0b743fd0e19c-kube-api-access-dfzt7\" (UniqueName: \"kubernetes.io/projected/0099729c-ec0f-4b6b-b375-0b743fd0e19c-kube-api-access-dfzt7\") pod \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") "
Nov 02 22:25:48.164461 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.164369    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-openshift-ca\") pod \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") "
Nov 02 22:25:48.164591 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.164577    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-xks-ca\") pod \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\" (UID: \"0099729c-ec0f-4b6b-b375-0b743fd0e19c\") "
Nov 02 22:25:48.164717 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.164692    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-openshift-ca" pod "0099729c-ec0f-4b6b-b375-0b743fd0e19c" (UID: "0099729c-ec0f-4b6b-b375-0b743fd0e19c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:48.165052 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.165033    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:48.166424 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.166398    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0099729c-ec0f-4b6b-b375-0b743fd0e19c-kube-api-access-dfzt7" pod "0099729c-ec0f-4b6b-b375-0b743fd0e19c" (UID: "0099729c-ec0f-4b6b-b375-0b743fd0e19c"). InnerVolumeSpecName "kube-api-access-dfzt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:48.166534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.166492    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-xks-ca" pod "0099729c-ec0f-4b6b-b375-0b743fd0e19c" (UID: "0099729c-ec0f-4b6b-b375-0b743fd0e19c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:48.266758 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.266727    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0099729c-ec0f-4b6b-b375-0b743fd0e19c-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:48.266758 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:48.266760    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfzt7\" (UniqueName: \"kubernetes.io/projected/0099729c-ec0f-4b6b-b375-0b743fd0e19c-kube-api-access-dfzt7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:49.001941 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:49.001915    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"
Nov 02 22:25:49.002120 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:49.001951    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst" event={"ID":"0099729c-ec0f-4b6b-b375-0b743fd0e19c","Type":"ContainerDied","Data":"1a1630c06f888542e9aebd8587db777d8df45e13101aae80557b2797f6d61a2a"}
Nov 02 22:25:49.002120 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:49.001995    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a1630c06f888542e9aebd8587db777d8df45e13101aae80557b2797f6d61a2a"
Nov 02 22:25:53.952477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.952445    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"]
Nov 02 22:25:53.953044 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953029    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f7d4c67d-1017-49a1-add1-0c40174a3427" containerName="payload-pre-processing"
Nov 02 22:25:53.953093 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953045    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d4c67d-1017-49a1-add1-0c40174a3427" containerName="payload-pre-processing"
Nov 02 22:25:53.953093 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953076    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6a7f7c74-dced-42c9-9240-a666235dec87" containerName="revoke-keys"
Nov 02 22:25:53.953093 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953081    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7f7c74-dced-42c9-9240-a666235dec87" containerName="revoke-keys"
Nov 02 22:25:53.953093 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953091    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0099729c-ec0f-4b6b-b375-0b743fd0e19c" containerName="revoke-keys"
Nov 02 22:25:53.953249 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953096    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0099729c-ec0f-4b6b-b375-0b743fd0e19c" containerName="revoke-keys"
Nov 02 22:25:53.953249 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953127    4528 cpu_manager.go:414] "containerMap: removing container" podUID="25830df0-4c4b-4d46-b821-f050dff8bd51" containerName="payload-processing"
Nov 02 22:25:53.953249 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953132    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25830df0-4c4b-4d46-b821-f050dff8bd51" containerName="payload-processing"
Nov 02 22:25:53.953249 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953215    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="25830df0-4c4b-4d46-b821-f050dff8bd51" containerName="payload-processing"
Nov 02 22:25:53.953249 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953226    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d4c67d-1017-49a1-add1-0c40174a3427" containerName="payload-pre-processing"
Nov 02 22:25:53.953249 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953238    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7f7c74-dced-42c9-9240-a666235dec87" containerName="revoke-keys"
Nov 02 22:25:53.953249 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.953245    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0099729c-ec0f-4b6b-b375-0b743fd0e19c" containerName="revoke-keys"
Nov 02 22:25:53.956755 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:53.956739    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.008618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.008583    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"]
Nov 02 22:25:54.041188 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.041152    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt7qm\" (UniqueName: \"kubernetes.io/projected/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-kube-api-access-jt7qm\") pod \"maas-api-revoke-sub-7aa13b47-gz9gc\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.041418 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.041395    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aa13b47-gz9gc\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.041554 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.041535    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aa13b47-gz9gc\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.054330 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.054297    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:25:54.054330 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.054324    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:25:54.054514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.054376    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:25:54.059166 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.059147    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:25:54.143070 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.143042    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt7qm\" (UniqueName: \"kubernetes.io/projected/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-kube-api-access-jt7qm\") pod \"maas-api-revoke-sub-7aa13b47-gz9gc\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.143250 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.143172    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aa13b47-gz9gc\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.143250 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.143242    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aa13b47-gz9gc\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.144078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.144054    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aa13b47-gz9gc\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.145526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.145506    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aa13b47-gz9gc\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.199609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.199580    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt7qm\" (UniqueName: \"kubernetes.io/projected/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-kube-api-access-jt7qm\") pod \"maas-api-revoke-sub-7aa13b47-gz9gc\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.269758 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.269735    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:54.638268 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:25:54.638228    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d047fc_f926_4100_9ac2_d1a5ea5bc266.slice/crio-a5dc55a977aa90c32597191df564522feb63f66c0adfee8f4357981297e26ca6 WatchSource:0}: Error finding container a5dc55a977aa90c32597191df564522feb63f66c0adfee8f4357981297e26ca6: Status 404 returned error can't find the container with id a5dc55a977aa90c32597191df564522feb63f66c0adfee8f4357981297e26ca6
Nov 02 22:25:54.640792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:54.640765    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"]
Nov 02 22:25:55.040903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:55.040866    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc" event={"ID":"c4d047fc-f926-4100-9ac2-d1a5ea5bc266","Type":"ContainerStarted","Data":"e56a4044ca3670c99876143bb5bbd2201aa887e86e9cc0161c268f01b0f29cd9"}
Nov 02 22:25:55.041320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:55.040924    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc" event={"ID":"c4d047fc-f926-4100-9ac2-d1a5ea5bc266","Type":"ContainerStarted","Data":"a5dc55a977aa90c32597191df564522feb63f66c0adfee8f4357981297e26ca6"}
Nov 02 22:25:55.155542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:55.155485    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc" podStartSLOduration=2.155468357 podStartE2EDuration="2.155468357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:55.154083125 +0000 UTC m=+4642.417619442" watchObservedRunningTime="2026-11-02 22:25:55.155468357 +0000 UTC m=+4642.419004673"
Nov 02 22:25:56.048596 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:56.048564    4528 generic.go:356] "Generic (PLEG): container finished" podID="c4d047fc-f926-4100-9ac2-d1a5ea5bc266" containerID="e56a4044ca3670c99876143bb5bbd2201aa887e86e9cc0161c268f01b0f29cd9" exitCode=0
Nov 02 22:25:56.048972 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:56.048647    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc" event={"ID":"c4d047fc-f926-4100-9ac2-d1a5ea5bc266","Type":"ContainerDied","Data":"e56a4044ca3670c99876143bb5bbd2201aa887e86e9cc0161c268f01b0f29cd9"}
Nov 02 22:25:57.203225 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.203203    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:57.281161 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.281133    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-kube-api-access-jt7qm\" (UniqueName: \"kubernetes.io/projected/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-kube-api-access-jt7qm\") pod \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") "
Nov 02 22:25:57.281300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.281239    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-xks-ca\") pod \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") "
Nov 02 22:25:57.281350 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.281334    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-openshift-ca\") pod \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\" (UID: \"c4d047fc-f926-4100-9ac2-d1a5ea5bc266\") "
Nov 02 22:25:57.281717 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.281687    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-openshift-ca" pod "c4d047fc-f926-4100-9ac2-d1a5ea5bc266" (UID: "c4d047fc-f926-4100-9ac2-d1a5ea5bc266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:25:57.281867 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.281847    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:57.283300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.283267    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-kube-api-access-jt7qm" pod "c4d047fc-f926-4100-9ac2-d1a5ea5bc266" (UID: "c4d047fc-f926-4100-9ac2-d1a5ea5bc266"). InnerVolumeSpecName "kube-api-access-jt7qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:25:57.283463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.283443    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-xks-ca" pod "c4d047fc-f926-4100-9ac2-d1a5ea5bc266" (UID: "c4d047fc-f926-4100-9ac2-d1a5ea5bc266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:25:57.382599 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.382516    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt7qm\" (UniqueName: \"kubernetes.io/projected/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-kube-api-access-jt7qm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:57.382599 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:57.382551    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d047fc-f926-4100-9ac2-d1a5ea5bc266-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:25:58.062698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:58.062663    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"
Nov 02 22:25:58.062875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:58.062706    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc" event={"ID":"c4d047fc-f926-4100-9ac2-d1a5ea5bc266","Type":"ContainerDied","Data":"a5dc55a977aa90c32597191df564522feb63f66c0adfee8f4357981297e26ca6"}
Nov 02 22:25:58.062875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:25:58.062757    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5dc55a977aa90c32597191df564522feb63f66c0adfee8f4357981297e26ca6"
Nov 02 22:26:06.982321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:06.982288    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"]
Nov 02 22:26:06.982887 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:06.982873    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c4d047fc-f926-4100-9ac2-d1a5ea5bc266" containerName="revoke-keys"
Nov 02 22:26:06.982929 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:06.982888    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d047fc-f926-4100-9ac2-d1a5ea5bc266" containerName="revoke-keys"
Nov 02 22:26:06.982984 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:06.982975    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d047fc-f926-4100-9ac2-d1a5ea5bc266" containerName="revoke-keys"
Nov 02 22:26:06.986858 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:06.986842    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.040511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.040475    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"]
Nov 02 22:26:07.085612 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.085577    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:26:07.085612 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.085585    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:26:07.085820 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.085638    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:26:07.091103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.091070    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:26:07.100817 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.100788    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrhx\" (UniqueName: \"kubernetes.io/projected/f8bc17e7-9227-48c2-8e88-b7631ded64dc-kube-api-access-fwrhx\") pod \"maas-api-revoke-sub-5dea93f8-t997m\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.100961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.100936    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea93f8-t997m\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.101510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.101488    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea93f8-t997m\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.203003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.202969    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea93f8-t997m\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.203215 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.203019    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrhx\" (UniqueName: \"kubernetes.io/projected/f8bc17e7-9227-48c2-8e88-b7631ded64dc-kube-api-access-fwrhx\") pod \"maas-api-revoke-sub-5dea93f8-t997m\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.203215 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.203066    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea93f8-t997m\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.203808 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.203784    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea93f8-t997m\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.205427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.205406    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea93f8-t997m\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.259599 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.259528    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrhx\" (UniqueName: \"kubernetes.io/projected/f8bc17e7-9227-48c2-8e88-b7631ded64dc-kube-api-access-fwrhx\") pod \"maas-api-revoke-sub-5dea93f8-t997m\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.302906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.302882    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:07.671527 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:26:07.671487    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bc17e7_9227_48c2_8e88_b7631ded64dc.slice/crio-9f6177567834ea87db82cd4a760c2c69fc14507acd3ef45bf451297401f29d16 WatchSource:0}: Error finding container 9f6177567834ea87db82cd4a760c2c69fc14507acd3ef45bf451297401f29d16: Status 404 returned error can't find the container with id 9f6177567834ea87db82cd4a760c2c69fc14507acd3ef45bf451297401f29d16
Nov 02 22:26:07.674884 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:07.674858    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"]
Nov 02 22:26:08.137175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:08.137138    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m" event={"ID":"f8bc17e7-9227-48c2-8e88-b7631ded64dc","Type":"ContainerStarted","Data":"cb3067305244d99ccee7feb3009aa3d52a80a7e07f139a0ce9e1679310608416"}
Nov 02 22:26:08.137573 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:08.137192    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m" event={"ID":"f8bc17e7-9227-48c2-8e88-b7631ded64dc","Type":"ContainerStarted","Data":"9f6177567834ea87db82cd4a760c2c69fc14507acd3ef45bf451297401f29d16"}
Nov 02 22:26:08.252586 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:08.252496    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m" podStartSLOduration=2.2524824 podStartE2EDuration="2.2524824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:08.249561526 +0000 UTC m=+4655.513097842" watchObservedRunningTime="2026-11-02 22:26:08.2524824 +0000 UTC m=+4655.516018712"
Nov 02 22:26:09.145303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:09.145267    4528 generic.go:356] "Generic (PLEG): container finished" podID="f8bc17e7-9227-48c2-8e88-b7631ded64dc" containerID="cb3067305244d99ccee7feb3009aa3d52a80a7e07f139a0ce9e1679310608416" exitCode=0
Nov 02 22:26:09.145761 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:09.145361    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m" event={"ID":"f8bc17e7-9227-48c2-8e88-b7631ded64dc","Type":"ContainerDied","Data":"cb3067305244d99ccee7feb3009aa3d52a80a7e07f139a0ce9e1679310608416"}
Nov 02 22:26:10.298040 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.298017    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:10.338273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.338246    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bc17e7-9227-48c2-8e88-b7631ded64dc-kube-api-access-fwrhx\" (UniqueName: \"kubernetes.io/projected/f8bc17e7-9227-48c2-8e88-b7631ded64dc-kube-api-access-fwrhx\") pod \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") "
Nov 02 22:26:10.338399 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.338283    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-xks-ca\") pod \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") "
Nov 02 22:26:10.338399 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.338312    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-openshift-ca\") pod \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\" (UID: \"f8bc17e7-9227-48c2-8e88-b7631ded64dc\") "
Nov 02 22:26:10.338685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.338664    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-openshift-ca" pod "f8bc17e7-9227-48c2-8e88-b7631ded64dc" (UID: "f8bc17e7-9227-48c2-8e88-b7631ded64dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:10.338840 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.338821    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:10.340342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.340314    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-xks-ca" pod "f8bc17e7-9227-48c2-8e88-b7631ded64dc" (UID: "f8bc17e7-9227-48c2-8e88-b7631ded64dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:10.340432 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.340365    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bc17e7-9227-48c2-8e88-b7631ded64dc-kube-api-access-fwrhx" pod "f8bc17e7-9227-48c2-8e88-b7631ded64dc" (UID: "f8bc17e7-9227-48c2-8e88-b7631ded64dc"). InnerVolumeSpecName "kube-api-access-fwrhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:10.440154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.440054    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrhx\" (UniqueName: \"kubernetes.io/projected/f8bc17e7-9227-48c2-8e88-b7631ded64dc-kube-api-access-fwrhx\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:10.440154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:10.440081    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8bc17e7-9227-48c2-8e88-b7631ded64dc-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:11.160251 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:11.160224    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"
Nov 02 22:26:11.160416 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:11.160220    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m" event={"ID":"f8bc17e7-9227-48c2-8e88-b7631ded64dc","Type":"ContainerDied","Data":"9f6177567834ea87db82cd4a760c2c69fc14507acd3ef45bf451297401f29d16"}
Nov 02 22:26:11.160416 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:11.160352    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f6177567834ea87db82cd4a760c2c69fc14507acd3ef45bf451297401f29d16"
Nov 02 22:26:14.113941 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:26:14.113901    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:26:25.183573 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:25.183537    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:26:38.290284 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:38.290252    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"]
Nov 02 22:26:38.290757 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:38.290493    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy" containerID="cri-o://ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da" gracePeriod=30
Nov 02 22:26:43.562513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.562448    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:26:43.639054 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639025    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-istio-token\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-istio-token\") pod \"2d310373-b874-4397-be84-c81975774f8d\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") "
Nov 02 22:26:43.639230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639062    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d310373-b874-4397-be84-c81975774f8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d310373-b874-4397-be84-c81975774f8d-istiod-ca-cert\") pod \"2d310373-b874-4397-be84-c81975774f8d\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") "
Nov 02 22:26:43.639230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639198    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-kube-api-access-ff2z2\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-kube-api-access-ff2z2\") pod \"2d310373-b874-4397-be84-c81975774f8d\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") "
Nov 02 22:26:43.639349 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639310    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-credential-socket\") pod \"2d310373-b874-4397-be84-c81975774f8d\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") "
Nov 02 22:26:43.639425 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639412    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-certs\") pod \"2d310373-b874-4397-be84-c81975774f8d\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") "
Nov 02 22:26:43.639485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639436    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d310373-b874-4397-be84-c81975774f8d-istiod-ca-cert" pod "2d310373-b874-4397-be84-c81975774f8d" (UID: "2d310373-b874-4397-be84-c81975774f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:43.639485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639444    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-envoy\") pod \"2d310373-b874-4397-be84-c81975774f8d\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") "
Nov 02 22:26:43.639611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639549    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-credential-socket" pod "2d310373-b874-4397-be84-c81975774f8d" (UID: "2d310373-b874-4397-be84-c81975774f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:43.639611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639600    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-socket\") pod \"2d310373-b874-4397-be84-c81975774f8d\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") "
Nov 02 22:26:43.639725 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639646    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-data\") pod \"2d310373-b874-4397-be84-c81975774f8d\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") "
Nov 02 22:26:43.639778 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639715    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-certs" pod "2d310373-b874-4397-be84-c81975774f8d" (UID: "2d310373-b874-4397-be84-c81975774f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:43.639778 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639730    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d310373-b874-4397-be84-c81975774f8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d310373-b874-4397-be84-c81975774f8d-istio-podinfo\") pod \"2d310373-b874-4397-be84-c81975774f8d\" (UID: \"2d310373-b874-4397-be84-c81975774f8d\") "
Nov 02 22:26:43.639871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639826    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-socket" pod "2d310373-b874-4397-be84-c81975774f8d" (UID: "2d310373-b874-4397-be84-c81975774f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:43.639951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.639931    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-data" pod "2d310373-b874-4397-be84-c81975774f8d" (UID: "2d310373-b874-4397-be84-c81975774f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:43.640293 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.640267    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:43.640411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.640295    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:43.640411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.640309    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d310373-b874-4397-be84-c81975774f8d-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:43.640411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.640324    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:43.640411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.640337    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:43.641414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.641390    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-istio-token" pod "2d310373-b874-4397-be84-c81975774f8d" (UID: "2d310373-b874-4397-be84-c81975774f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:43.641507 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.641411    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-kube-api-access-ff2z2" pod "2d310373-b874-4397-be84-c81975774f8d" (UID: "2d310373-b874-4397-be84-c81975774f8d"). InnerVolumeSpecName "kube-api-access-ff2z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:43.641671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.641652    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-envoy" pod "2d310373-b874-4397-be84-c81975774f8d" (UID: "2d310373-b874-4397-be84-c81975774f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:26:43.641740 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.641722    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d310373-b874-4397-be84-c81975774f8d-istio-podinfo" pod "2d310373-b874-4397-be84-c81975774f8d" (UID: "2d310373-b874-4397-be84-c81975774f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:26:43.741716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.741688    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d310373-b874-4397-be84-c81975774f8d-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:43.741716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.741713    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:43.741716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.741723    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2z2\" (UniqueName: \"kubernetes.io/projected/2d310373-b874-4397-be84-c81975774f8d-kube-api-access-ff2z2\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:43.741716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:43.741731    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d310373-b874-4397-be84-c81975774f8d-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:44.112829 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:26:44.112796    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:26:44.390864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.390772    4528 generic.go:356] "Generic (PLEG): container finished" podID="2d310373-b874-4397-be84-c81975774f8d" containerID="ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da" exitCode=0
Nov 02 22:26:44.390864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.390819    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"
Nov 02 22:26:44.390864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.390853    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" event={"ID":"2d310373-b874-4397-be84-c81975774f8d","Type":"ContainerDied","Data":"ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da"}
Nov 02 22:26:44.391143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.390906    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v" event={"ID":"2d310373-b874-4397-be84-c81975774f8d","Type":"ContainerDied","Data":"32e17950443ef6d854d0f37f4fbf55a1922bec3b4a21c59d4382140273a0e72c"}
Nov 02 22:26:44.391143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.390931    4528 scope.go:122] "RemoveContainer" containerID="ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da"
Nov 02 22:26:44.401555 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.401539    4528 scope.go:122] "RemoveContainer" containerID="ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da"
Nov 02 22:26:44.401836 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:26:44.401820    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da\": container with ID starting with ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da not found: ID does not exist" containerID="ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da"
Nov 02 22:26:44.401883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.401843    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da"} err="failed to get container status \"ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da\": rpc error: code = NotFound desc = could not find container \"ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da\": container with ID starting with ef845fd29433bfef676b5b318ae76943ed0ea9307b0d32f4e511c8f0350fc5da not found: ID does not exist"
Nov 02 22:26:44.606427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.606397    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"]
Nov 02 22:26:44.656630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.656561    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hbd6v"]
Nov 02 22:26:44.913803 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:44.913727    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d310373-b874-4397-be84-c81975774f8d" path="/var/lib/kubelet/pods/2d310373-b874-4397-be84-c81975774f8d/volumes"
Nov 02 22:26:53.316696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:53.316651    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"]
Nov 02 22:26:53.422237 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:53.422196    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"]
Nov 02 22:26:56.175876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.175836    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"]
Nov 02 22:26:56.176467 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.176452    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy"
Nov 02 22:26:56.176517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.176467    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy"
Nov 02 22:26:56.176517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.176490    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f8bc17e7-9227-48c2-8e88-b7631ded64dc" containerName="revoke-keys"
Nov 02 22:26:56.176517 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.176496    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bc17e7-9227-48c2-8e88-b7631ded64dc" containerName="revoke-keys"
Nov 02 22:26:56.176615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.176589    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d310373-b874-4397-be84-c81975774f8d" containerName="istio-proxy"
Nov 02 22:26:56.176615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.176602    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bc17e7-9227-48c2-8e88-b7631ded64dc" containerName="revoke-keys"
Nov 02 22:26:56.179911 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.179896    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.231893 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.231862    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"]
Nov 02 22:26:56.235190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.235149    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a4f635e-f8n8q\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.235348 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.235223    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78plh\" (UniqueName: \"kubernetes.io/projected/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-kube-api-access-78plh\") pod \"maas-api-revoke-sub-6a4f635e-f8n8q\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.235348 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.235299    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a4f635e-f8n8q\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.279696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.279665    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:26:56.279877 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.279698    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:26:56.279877 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.279698    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:26:56.283502 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.283485    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:26:56.336435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.336403    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a4f635e-f8n8q\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.336618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.336460    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78plh\" (UniqueName: \"kubernetes.io/projected/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-kube-api-access-78plh\") pod \"maas-api-revoke-sub-6a4f635e-f8n8q\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.336618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.336477    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a4f635e-f8n8q\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.337240 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.337218    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a4f635e-f8n8q\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.338970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.338940    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a4f635e-f8n8q\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.393434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.393405    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78plh\" (UniqueName: \"kubernetes.io/projected/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-kube-api-access-78plh\") pod \"maas-api-revoke-sub-6a4f635e-f8n8q\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.493439 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.493369    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:56.862664 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:26:56.862627    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a570bd_bef8_4a8c_a49a_0615dfe1bddf.slice/crio-7116f9afa9c1453f621cb45f9ff8e5bb11f5d7a1711c8f608b990f49cd15cd97 WatchSource:0}: Error finding container 7116f9afa9c1453f621cb45f9ff8e5bb11f5d7a1711c8f608b990f49cd15cd97: Status 404 returned error can't find the container with id 7116f9afa9c1453f621cb45f9ff8e5bb11f5d7a1711c8f608b990f49cd15cd97
Nov 02 22:26:56.867077 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:56.867056    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"]
Nov 02 22:26:57.500084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:57.499973    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q" event={"ID":"55a570bd-bef8-4a8c-a49a-0615dfe1bddf","Type":"ContainerStarted","Data":"f4e604a1f09fb5d9a3ad74b7c8f3484fc610665866d0246b71f25fd48ac2071a"}
Nov 02 22:26:57.500084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:57.500069    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q" event={"ID":"55a570bd-bef8-4a8c-a49a-0615dfe1bddf","Type":"ContainerStarted","Data":"7116f9afa9c1453f621cb45f9ff8e5bb11f5d7a1711c8f608b990f49cd15cd97"}
Nov 02 22:26:57.612778 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:57.612725    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q" podStartSLOduration=1.6127124959999999 podStartE2EDuration="1.612712496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:57.612142289 +0000 UTC m=+4704.875678607" watchObservedRunningTime="2026-11-02 22:26:57.612712496 +0000 UTC m=+4704.876248812"
Nov 02 22:26:58.317823 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.317767    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh" podUID="2f1d0b84-23f8-4ffe-be30-9f71cbc95655" containerName="payload-processing" containerID="cri-o://0ea385dd535228c729c2e717f338217de1c905ce1a76b3822853f49a40767e2b" gracePeriod=25
Nov 02 22:26:58.423149 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.423102    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql" podUID="11674de5-804a-4164-8838-f06115b70e7f" containerName="payload-pre-processing" containerID="cri-o://b378d96dafe50ca38b99feec624f672ac418f9bc106e4d35988b80cb8d991542" gracePeriod=25
Nov 02 22:26:58.508522 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.508487    4528 generic.go:356] "Generic (PLEG): container finished" podID="2f1d0b84-23f8-4ffe-be30-9f71cbc95655" containerID="0ea385dd535228c729c2e717f338217de1c905ce1a76b3822853f49a40767e2b" exitCode=0
Nov 02 22:26:58.508947 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.508556    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh" event={"ID":"2f1d0b84-23f8-4ffe-be30-9f71cbc95655","Type":"ContainerDied","Data":"0ea385dd535228c729c2e717f338217de1c905ce1a76b3822853f49a40767e2b"}
Nov 02 22:26:58.510556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.510532    4528 generic.go:356] "Generic (PLEG): container finished" podID="55a570bd-bef8-4a8c-a49a-0615dfe1bddf" containerID="f4e604a1f09fb5d9a3ad74b7c8f3484fc610665866d0246b71f25fd48ac2071a" exitCode=0
Nov 02 22:26:58.510696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.510583    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q" event={"ID":"55a570bd-bef8-4a8c-a49a-0615dfe1bddf","Type":"ContainerDied","Data":"f4e604a1f09fb5d9a3ad74b7c8f3484fc610665866d0246b71f25fd48ac2071a"}
Nov 02 22:26:58.597636 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.597612    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:26:58.662523 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.662492    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-kube-api-access-74pkh\" (UniqueName: \"kubernetes.io/projected/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-kube-api-access-74pkh\") pod \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\" (UID: \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\") "
Nov 02 22:26:58.662680 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.662625    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-plugins-config-volume\") pod \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\" (UID: \"2f1d0b84-23f8-4ffe-be30-9f71cbc95655\") "
Nov 02 22:26:58.662976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.662952    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-plugins-config-volume" pod "2f1d0b84-23f8-4ffe-be30-9f71cbc95655" (UID: "2f1d0b84-23f8-4ffe-be30-9f71cbc95655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:58.664552 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.664526    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-kube-api-access-74pkh" pod "2f1d0b84-23f8-4ffe-be30-9f71cbc95655" (UID: "2f1d0b84-23f8-4ffe-be30-9f71cbc95655"). InnerVolumeSpecName "kube-api-access-74pkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:58.764842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.764808    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74pkh\" (UniqueName: \"kubernetes.io/projected/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-kube-api-access-74pkh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:58.764842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:58.764844    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1d0b84-23f8-4ffe-be30-9f71cbc95655-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:59.518426 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.518396    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"
Nov 02 22:26:59.518820 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.518430    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh" event={"ID":"2f1d0b84-23f8-4ffe-be30-9f71cbc95655","Type":"ContainerDied","Data":"ca2196885994453733baceb1656a0f5fc5a65b6edf1457d8eeddaf0d206906a3"}
Nov 02 22:26:59.518820 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.518492    4528 scope.go:122] "RemoveContainer" containerID="0ea385dd535228c729c2e717f338217de1c905ce1a76b3822853f49a40767e2b"
Nov 02 22:26:59.666951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.666926    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:26:59.683426 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.683401    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"]
Nov 02 22:26:59.733294 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.733269    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dd28e6-d5b76cf5f-bmcgh"]
Nov 02 22:26:59.777259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.777201    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-openshift-ca\") pod \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") "
Nov 02 22:26:59.777370 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.777295    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-xks-ca\") pod \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") "
Nov 02 22:26:59.777370 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.777333    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-kube-api-access-78plh\" (UniqueName: \"kubernetes.io/projected/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-kube-api-access-78plh\") pod \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\" (UID: \"55a570bd-bef8-4a8c-a49a-0615dfe1bddf\") "
Nov 02 22:26:59.777582 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.777557    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-openshift-ca" pod "55a570bd-bef8-4a8c-a49a-0615dfe1bddf" (UID: "55a570bd-bef8-4a8c-a49a-0615dfe1bddf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:26:59.777901 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.777876    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:59.779359 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.779339    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-xks-ca" pod "55a570bd-bef8-4a8c-a49a-0615dfe1bddf" (UID: "55a570bd-bef8-4a8c-a49a-0615dfe1bddf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:26:59.779461 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.779443    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-kube-api-access-78plh" pod "55a570bd-bef8-4a8c-a49a-0615dfe1bddf" (UID: "55a570bd-bef8-4a8c-a49a-0615dfe1bddf"). InnerVolumeSpecName "kube-api-access-78plh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:26:59.879346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.879315    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:26:59.879346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:26:59.879341    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78plh\" (UniqueName: \"kubernetes.io/projected/55a570bd-bef8-4a8c-a49a-0615dfe1bddf-kube-api-access-78plh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:00.525974 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:00.525948    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"
Nov 02 22:27:00.526454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:00.525982    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q" event={"ID":"55a570bd-bef8-4a8c-a49a-0615dfe1bddf","Type":"ContainerDied","Data":"7116f9afa9c1453f621cb45f9ff8e5bb11f5d7a1711c8f608b990f49cd15cd97"}
Nov 02 22:27:00.526454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:00.526030    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7116f9afa9c1453f621cb45f9ff8e5bb11f5d7a1711c8f608b990f49cd15cd97"
Nov 02 22:27:00.917339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:00.917258    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1d0b84-23f8-4ffe-be30-9f71cbc95655" path="/var/lib/kubelet/pods/2f1d0b84-23f8-4ffe-be30-9f71cbc95655/volumes"
Nov 02 22:27:02.361447 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:02.361406    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql" podUID="11674de5-804a-4164-8838-f06115b70e7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:27:05.447175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:05.447139    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"]
Nov 02 22:27:05.447566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:05.447364    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy" containerID="cri-o://cae172c79e766a2f0651a8f02b1486583ddedc7db7b115e151ef6d9478425ff2" gracePeriod=30
Nov 02 22:27:07.034961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.034921    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"]
Nov 02 22:27:07.035505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.035490    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2f1d0b84-23f8-4ffe-be30-9f71cbc95655" containerName="payload-processing"
Nov 02 22:27:07.035505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.035504    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1d0b84-23f8-4ffe-be30-9f71cbc95655" containerName="payload-processing"
Nov 02 22:27:07.035630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.035516    4528 cpu_manager.go:414] "containerMap: removing container" podUID="55a570bd-bef8-4a8c-a49a-0615dfe1bddf" containerName="revoke-keys"
Nov 02 22:27:07.035630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.035521    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a570bd-bef8-4a8c-a49a-0615dfe1bddf" containerName="revoke-keys"
Nov 02 22:27:07.035738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.035636    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1d0b84-23f8-4ffe-be30-9f71cbc95655" containerName="payload-processing"
Nov 02 22:27:07.035738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.035645    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a570bd-bef8-4a8c-a49a-0615dfe1bddf" containerName="revoke-keys"
Nov 02 22:27:07.039391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.039367    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.089744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.089709    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"]
Nov 02 22:27:07.136804 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.136772    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:27:07.136804 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.136781    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:27:07.136804 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.136781    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:27:07.142651 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.142634    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:27:07.158943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.158919    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461995c1-ndfgp\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.159047 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.158959    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461995c1-ndfgp\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.159096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.159084    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtzdv\" (UniqueName: \"kubernetes.io/projected/eeb6bac5-7211-44ed-9f19-e159c4037568-kube-api-access-qtzdv\") pod \"maas-api-revoke-sub-461995c1-ndfgp\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.260459 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.260434    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461995c1-ndfgp\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.260604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.260474    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461995c1-ndfgp\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.260651 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.260631    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzdv\" (UniqueName: \"kubernetes.io/projected/eeb6bac5-7211-44ed-9f19-e159c4037568-kube-api-access-qtzdv\") pod \"maas-api-revoke-sub-461995c1-ndfgp\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.261055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.261034    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461995c1-ndfgp\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.262731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.262712    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461995c1-ndfgp\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.319856 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.319794    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzdv\" (UniqueName: \"kubernetes.io/projected/eeb6bac5-7211-44ed-9f19-e159c4037568-kube-api-access-qtzdv\") pod \"maas-api-revoke-sub-461995c1-ndfgp\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.352413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.352387    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:07.722061 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:27:07.722028    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb6bac5_7211_44ed_9f19_e159c4037568.slice/crio-4a450db788141914d16217221fe676891a7f016dea959de448226b3a0072ee3b WatchSource:0}: Error finding container 4a450db788141914d16217221fe676891a7f016dea959de448226b3a0072ee3b: Status 404 returned error can't find the container with id 4a450db788141914d16217221fe676891a7f016dea959de448226b3a0072ee3b
Nov 02 22:27:07.725295 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:07.725268    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"]
Nov 02 22:27:08.587999 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:08.587962    4528 generic.go:356] "Generic (PLEG): container finished" podID="eeb6bac5-7211-44ed-9f19-e159c4037568" containerID="c3797d95cd9fee605d47a2b8c282cdac36fb717d6f876fd36d28bb85a9d8dc86" exitCode=0
Nov 02 22:27:08.588407 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:08.588052    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp" event={"ID":"eeb6bac5-7211-44ed-9f19-e159c4037568","Type":"ContainerDied","Data":"c3797d95cd9fee605d47a2b8c282cdac36fb717d6f876fd36d28bb85a9d8dc86"}
Nov 02 22:27:08.588407 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:08.588141    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp" event={"ID":"eeb6bac5-7211-44ed-9f19-e159c4037568","Type":"ContainerStarted","Data":"4a450db788141914d16217221fe676891a7f016dea959de448226b3a0072ee3b"}
Nov 02 22:27:09.225264 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.225229    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:27:09.749848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.749812    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:09.825143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.823531    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-xks-ca\") pod \"eeb6bac5-7211-44ed-9f19-e159c4037568\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") "
Nov 02 22:27:09.825143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.823653    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-openshift-ca\") pod \"eeb6bac5-7211-44ed-9f19-e159c4037568\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") "
Nov 02 22:27:09.825143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.823793    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb6bac5-7211-44ed-9f19-e159c4037568-kube-api-access-qtzdv\" (UniqueName: \"kubernetes.io/projected/eeb6bac5-7211-44ed-9f19-e159c4037568-kube-api-access-qtzdv\") pod \"eeb6bac5-7211-44ed-9f19-e159c4037568\" (UID: \"eeb6bac5-7211-44ed-9f19-e159c4037568\") "
Nov 02 22:27:09.848539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.848439    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-openshift-ca" pod "eeb6bac5-7211-44ed-9f19-e159c4037568" (UID: "eeb6bac5-7211-44ed-9f19-e159c4037568"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:09.860596 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.860540    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb6bac5-7211-44ed-9f19-e159c4037568-kube-api-access-qtzdv" pod "eeb6bac5-7211-44ed-9f19-e159c4037568" (UID: "eeb6bac5-7211-44ed-9f19-e159c4037568"). InnerVolumeSpecName "kube-api-access-qtzdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:09.874140 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.871383    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-xks-ca" pod "eeb6bac5-7211-44ed-9f19-e159c4037568" (UID: "eeb6bac5-7211-44ed-9f19-e159c4037568"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:27:09.931162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.931104    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtzdv\" (UniqueName: \"kubernetes.io/projected/eeb6bac5-7211-44ed-9f19-e159c4037568-kube-api-access-qtzdv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:09.931162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.931161    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:09.931439 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:09.931177    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bac5-7211-44ed-9f19-e159c4037568-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.604657 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.604611    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"
Nov 02 22:27:10.604848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.604613    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp" event={"ID":"eeb6bac5-7211-44ed-9f19-e159c4037568","Type":"ContainerDied","Data":"4a450db788141914d16217221fe676891a7f016dea959de448226b3a0072ee3b"}
Nov 02 22:27:10.604933 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.604888    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a450db788141914d16217221fe676891a7f016dea959de448226b3a0072ee3b"
Nov 02 22:27:10.607498 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.607474    4528 generic.go:356] "Generic (PLEG): container finished" podID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerID="cae172c79e766a2f0651a8f02b1486583ddedc7db7b115e151ef6d9478425ff2" exitCode=0
Nov 02 22:27:10.607648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.607518    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" event={"ID":"b183f95a-965b-4052-b80f-f7fb5604a50c","Type":"ContainerDied","Data":"cae172c79e766a2f0651a8f02b1486583ddedc7db7b115e151ef6d9478425ff2"}
Nov 02 22:27:10.607648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.607563    4528 scope.go:122] "RemoveContainer" containerID="07a034111a82ee829065e2f29be953689c3d34ffd4323f60967914000a71ac2d"
Nov 02 22:27:10.703903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.703878    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:27:10.844137 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844090    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-socket\") pod \"b183f95a-965b-4052-b80f-f7fb5604a50c\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") "
Nov 02 22:27:10.844692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844389    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-socket" pod "b183f95a-965b-4052-b80f-f7fb5604a50c" (UID: "b183f95a-965b-4052-b80f-f7fb5604a50c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:10.844692 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844688    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-credential-socket\") pod \"b183f95a-965b-4052-b80f-f7fb5604a50c\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") "
Nov 02 22:27:10.844889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844718    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-kube-api-access-strbv\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-kube-api-access-strbv\") pod \"b183f95a-965b-4052-b80f-f7fb5604a50c\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") "
Nov 02 22:27:10.844889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844757    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b183f95a-965b-4052-b80f-f7fb5604a50c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b183f95a-965b-4052-b80f-f7fb5604a50c-istiod-ca-cert\") pod \"b183f95a-965b-4052-b80f-f7fb5604a50c\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") "
Nov 02 22:27:10.844889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844806    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-podinfo\") pod \"b183f95a-965b-4052-b80f-f7fb5604a50c\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") "
Nov 02 22:27:10.845069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844927    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-envoy\") pod \"b183f95a-965b-4052-b80f-f7fb5604a50c\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") "
Nov 02 22:27:10.845069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844948    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-certs\") pod \"b183f95a-965b-4052-b80f-f7fb5604a50c\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") "
Nov 02 22:27:10.845069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844966    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-credential-socket" pod "b183f95a-965b-4052-b80f-f7fb5604a50c" (UID: "b183f95a-965b-4052-b80f-f7fb5604a50c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:10.845069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.844991    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-token\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-token\") pod \"b183f95a-965b-4052-b80f-f7fb5604a50c\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") "
Nov 02 22:27:10.845069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.845047    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-data\") pod \"b183f95a-965b-4052-b80f-f7fb5604a50c\" (UID: \"b183f95a-965b-4052-b80f-f7fb5604a50c\") "
Nov 02 22:27:10.845362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.845202    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b183f95a-965b-4052-b80f-f7fb5604a50c-istiod-ca-cert" pod "b183f95a-965b-4052-b80f-f7fb5604a50c" (UID: "b183f95a-965b-4052-b80f-f7fb5604a50c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:10.845470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.845426    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-data" pod "b183f95a-965b-4052-b80f-f7fb5604a50c" (UID: "b183f95a-965b-4052-b80f-f7fb5604a50c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:10.845601 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.845549    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-certs" pod "b183f95a-965b-4052-b80f-f7fb5604a50c" (UID: "b183f95a-965b-4052-b80f-f7fb5604a50c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:10.845746 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.845730    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.845814 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.845748    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.845814 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.845757    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.845814 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.845767    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.845814 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.845780    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b183f95a-965b-4052-b80f-f7fb5604a50c-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.847674 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.847649    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-podinfo" pod "b183f95a-965b-4052-b80f-f7fb5604a50c" (UID: "b183f95a-965b-4052-b80f-f7fb5604a50c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:27:10.847785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.847675    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-envoy" pod "b183f95a-965b-4052-b80f-f7fb5604a50c" (UID: "b183f95a-965b-4052-b80f-f7fb5604a50c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:10.848010 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.847990    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-kube-api-access-strbv" pod "b183f95a-965b-4052-b80f-f7fb5604a50c" (UID: "b183f95a-965b-4052-b80f-f7fb5604a50c"). InnerVolumeSpecName "kube-api-access-strbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:10.849969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.848257    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-token" pod "b183f95a-965b-4052-b80f-f7fb5604a50c" (UID: "b183f95a-965b-4052-b80f-f7fb5604a50c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:10.947081 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.947055    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.947286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.947085    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.947286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.947100    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-strbv\" (UniqueName: \"kubernetes.io/projected/b183f95a-965b-4052-b80f-f7fb5604a50c-kube-api-access-strbv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:10.947286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:10.947145    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b183f95a-965b-4052-b80f-f7fb5604a50c-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:11.173944 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:11.173910    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"]
Nov 02 22:27:11.177267 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:11.177248    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b5e8794c-84w2k"]
Nov 02 22:27:11.339422 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:11.339388    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"]
Nov 02 22:27:11.339666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:11.339637    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy" containerID="cri-o://33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b" gracePeriod=30
Nov 02 22:27:11.615957 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:11.615916    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745" event={"ID":"b183f95a-965b-4052-b80f-f7fb5604a50c","Type":"ContainerDied","Data":"8542fb66eb89ec825c0d7ed5ecc3f1939b19048f54f45839710dc23b5a2553a7"}
Nov 02 22:27:11.615957 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:11.615949    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"
Nov 02 22:27:11.616231 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:11.615987    4528 scope.go:122] "RemoveContainer" containerID="cae172c79e766a2f0651a8f02b1486583ddedc7db7b115e151ef6d9478425ff2"
Nov 02 22:27:11.781922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:11.781889    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"]
Nov 02 22:27:11.785666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:11.785644    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3edc8c-openshift-default-85487f77f7-sh745"]
Nov 02 22:27:12.342554 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:12.342514    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql" podUID="11674de5-804a-4164-8838-f06115b70e7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:27:12.914184 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:12.914153    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7f7c74-dced-42c9-9240-a666235dec87" path="/var/lib/kubelet/pods/6a7f7c74-dced-42c9-9240-a666235dec87/volumes"
Nov 02 22:27:12.914655 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:12.914641    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" path="/var/lib/kubelet/pods/b183f95a-965b-4052-b80f-f7fb5604a50c/volumes"
Nov 02 22:27:14.113497 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:27:14.113461    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:27:16.613482 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.613418    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:27:16.649394 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.649358    4528 generic.go:356] "Generic (PLEG): container finished" podID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerID="33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b" exitCode=0
Nov 02 22:27:16.649553 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.649414    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"
Nov 02 22:27:16.649553 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.649415    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" event={"ID":"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a","Type":"ContainerDied","Data":"33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b"}
Nov 02 22:27:16.649553 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.649538    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc" event={"ID":"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a","Type":"ContainerDied","Data":"c595d07d1471af19b38c2ee34bbcd56d8a96954026649fa162459d057c5e9dac"}
Nov 02 22:27:16.649688 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.649563    4528 scope.go:122] "RemoveContainer" containerID="33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b"
Nov 02 22:27:16.658913 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.658897    4528 scope.go:122] "RemoveContainer" containerID="0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139"
Nov 02 22:27:16.667347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.667332    4528 scope.go:122] "RemoveContainer" containerID="33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b"
Nov 02 22:27:16.667609 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:27:16.667589    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b\": container with ID starting with 33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b not found: ID does not exist" containerID="33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b"
Nov 02 22:27:16.667684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.667616    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b"} err="failed to get container status \"33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b\": rpc error: code = NotFound desc = could not find container \"33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b\": container with ID starting with 33d23c7f42d5c73763835221d407b30fe697465d1705e7bafcc86406f1fa0c0b not found: ID does not exist"
Nov 02 22:27:16.667684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.667634    4528 scope.go:122] "RemoveContainer" containerID="0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139"
Nov 02 22:27:16.667883 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:27:16.667866    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139\": container with ID starting with 0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139 not found: ID does not exist" containerID="0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139"
Nov 02 22:27:16.667928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.667889    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139"} err="failed to get container status \"0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139\": rpc error: code = NotFound desc = could not find container \"0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139\": container with ID starting with 0acef846395c12485f839a0f0f104faf067e10d838124b190233c133b3fd0139 not found: ID does not exist"
Nov 02 22:27:16.717100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717076    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istiod-ca-cert\") pod \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") "
Nov 02 22:27:16.717239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717190    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-envoy\") pod \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") "
Nov 02 22:27:16.717239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717214    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-credential-socket\") pod \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") "
Nov 02 22:27:16.717351 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717246    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-data\") pod \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") "
Nov 02 22:27:16.717351 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717318    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-certs\") pod \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") "
Nov 02 22:27:16.717451 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717351    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-kube-api-access-pn559\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-kube-api-access-pn559\") pod \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") "
Nov 02 22:27:16.717451 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717375    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-podinfo\") pod \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") "
Nov 02 22:27:16.717451 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717433    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-token\") pod \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") "
Nov 02 22:27:16.717610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717475    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-socket\") pod \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\" (UID: \"a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a\") "
Nov 02 22:27:16.717610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717508    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-credential-socket" pod "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" (UID: "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:16.717610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717563    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istiod-ca-cert" pod "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" (UID: "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:16.717610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717583    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-data" pod "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" (UID: "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:16.717861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717836    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-certs" pod "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" (UID: "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:16.717861 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.717849    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-socket" pod "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" (UID: "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:16.718022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.718004    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:16.718096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.718028    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:16.718096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.718041    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:16.718096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.718055    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:16.718096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.718071    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:16.719438 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.719413    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-envoy" pod "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" (UID: "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:27:16.719530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.719502    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-kube-api-access-pn559" pod "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" (UID: "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a"). InnerVolumeSpecName "kube-api-access-pn559". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:16.719608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.719585    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-podinfo" pod "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" (UID: "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:27:16.719662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.719632    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-token" pod "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" (UID: "a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:16.819539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.819502    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn559\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-kube-api-access-pn559\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:16.819539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.819529    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:16.819539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.819540    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:16.819539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:16.819547    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:17.064779 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:17.064746    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"]
Nov 02 22:27:17.070607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:17.070580    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b5e8794c-openshift-default-7d7489bdd5-8f9wc"]
Nov 02 22:27:18.664578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.664543    4528 generic.go:356] "Generic (PLEG): container finished" podID="11674de5-804a-4164-8838-f06115b70e7f" containerID="b378d96dafe50ca38b99feec624f672ac418f9bc106e4d35988b80cb8d991542" exitCode=0
Nov 02 22:27:18.664971 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.664615    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql" event={"ID":"11674de5-804a-4164-8838-f06115b70e7f","Type":"ContainerDied","Data":"b378d96dafe50ca38b99feec624f672ac418f9bc106e4d35988b80cb8d991542"}
Nov 02 22:27:18.664971 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.664718    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql" event={"ID":"11674de5-804a-4164-8838-f06115b70e7f","Type":"ContainerDied","Data":"08f8a680d726434f7e6466e941463ade0cf8226a29c98a70ee162a60c60f220b"}
Nov 02 22:27:18.664971 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.664745    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08f8a680d726434f7e6466e941463ade0cf8226a29c98a70ee162a60c60f220b"
Nov 02 22:27:18.684051 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.684000    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:27:18.687983 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.687963    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"]
Nov 02 22:27:18.688511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688498    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy"
Nov 02 22:27:18.688566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688511    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy"
Nov 02 22:27:18.688566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688526    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy"
Nov 02 22:27:18.688566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688532    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy"
Nov 02 22:27:18.688566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688546    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy"
Nov 02 22:27:18.688566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688550    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy"
Nov 02 22:27:18.688566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688561    4528 cpu_manager.go:414] "containerMap: removing container" podUID="11674de5-804a-4164-8838-f06115b70e7f" containerName="payload-pre-processing"
Nov 02 22:27:18.688566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688566    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11674de5-804a-4164-8838-f06115b70e7f" containerName="payload-pre-processing"
Nov 02 22:27:18.688790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688576    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy"
Nov 02 22:27:18.688790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688581    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy"
Nov 02 22:27:18.688790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688590    4528 cpu_manager.go:414] "containerMap: removing container" podUID="eeb6bac5-7211-44ed-9f19-e159c4037568" containerName="revoke-keys"
Nov 02 22:27:18.688790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688595    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb6bac5-7211-44ed-9f19-e159c4037568" containerName="revoke-keys"
Nov 02 22:27:18.688790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688674    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="11674de5-804a-4164-8838-f06115b70e7f" containerName="payload-pre-processing"
Nov 02 22:27:18.688790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688684    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy"
Nov 02 22:27:18.688790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688690    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb6bac5-7211-44ed-9f19-e159c4037568" containerName="revoke-keys"
Nov 02 22:27:18.688790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.688699    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy"
Nov 02 22:27:18.691851 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.691836    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:18.745057 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.745031    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:27:18.745200 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.745094    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:27:18.786970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.786937    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:27:18.787144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.787041    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:27:18.790283 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.790257    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"]
Nov 02 22:27:18.839752 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.839722    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11674de5-804a-4164-8838-f06115b70e7f-kube-api-access-26fgs\" (UniqueName: \"kubernetes.io/projected/11674de5-804a-4164-8838-f06115b70e7f-kube-api-access-26fgs\") pod \"11674de5-804a-4164-8838-f06115b70e7f\" (UID: \"11674de5-804a-4164-8838-f06115b70e7f\") "
Nov 02 22:27:18.839909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.839860    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11674de5-804a-4164-8838-f06115b70e7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11674de5-804a-4164-8838-f06115b70e7f-plugins-config-volume\") pod \"11674de5-804a-4164-8838-f06115b70e7f\" (UID: \"11674de5-804a-4164-8838-f06115b70e7f\") "
Nov 02 22:27:18.840152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.840126    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wnq\" (UniqueName: \"kubernetes.io/projected/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-kube-api-access-s7wnq\") pod \"maas-api-revoke-sub-49d37bd0-p5d8t\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:18.840322 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.840295    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d37bd0-p5d8t\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:18.840322 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.840301    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11674de5-804a-4164-8838-f06115b70e7f-plugins-config-volume" pod "11674de5-804a-4164-8838-f06115b70e7f" (UID: "11674de5-804a-4164-8838-f06115b70e7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:18.840533 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.840380    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d37bd0-p5d8t\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:18.840533 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.840531    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11674de5-804a-4164-8838-f06115b70e7f-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:18.842002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.841974    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11674de5-804a-4164-8838-f06115b70e7f-kube-api-access-26fgs" pod "11674de5-804a-4164-8838-f06115b70e7f" (UID: "11674de5-804a-4164-8838-f06115b70e7f"). InnerVolumeSpecName "kube-api-access-26fgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:18.915452 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.915418    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" path="/var/lib/kubelet/pods/a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a/volumes"
Nov 02 22:27:18.942147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.942081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d37bd0-p5d8t\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:18.942267 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.942172    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d37bd0-p5d8t\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:18.942315 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.942278    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wnq\" (UniqueName: \"kubernetes.io/projected/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-kube-api-access-s7wnq\") pod \"maas-api-revoke-sub-49d37bd0-p5d8t\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:18.942352 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.942326    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fgs\" (UniqueName: \"kubernetes.io/projected/11674de5-804a-4164-8838-f06115b70e7f-kube-api-access-26fgs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:18.942769 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.942744    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d37bd0-p5d8t\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:18.944316 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.944293    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d37bd0-p5d8t\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:18.999626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:18.999596    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wnq\" (UniqueName: \"kubernetes.io/projected/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-kube-api-access-s7wnq\") pod \"maas-api-revoke-sub-49d37bd0-p5d8t\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:19.005172 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:19.005151    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:19.375282 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:27:19.375242    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f71b322_9fb8_4f6c_8662_48c5fb7ee383.slice/crio-e05094c017aefb3b0b9bedfd62c48f5497de1e641d9fcac74f152cc0ceaec6e8 WatchSource:0}: Error finding container e05094c017aefb3b0b9bedfd62c48f5497de1e641d9fcac74f152cc0ceaec6e8: Status 404 returned error can't find the container with id e05094c017aefb3b0b9bedfd62c48f5497de1e641d9fcac74f152cc0ceaec6e8
Nov 02 22:27:19.377630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:19.377600    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"]
Nov 02 22:27:19.442331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:19.442305    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"]
Nov 02 22:27:19.539156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:19.539125    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"]
Nov 02 22:27:19.672751 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:19.672670    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t" event={"ID":"7f71b322-9fb8-4f6c-8662-48c5fb7ee383","Type":"ContainerStarted","Data":"e05094c017aefb3b0b9bedfd62c48f5497de1e641d9fcac74f152cc0ceaec6e8"}
Nov 02 22:27:19.673194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:19.672754    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"
Nov 02 22:27:19.841270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:19.841245    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"]
Nov 02 22:27:19.844256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:19.844222    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dd28e6-7b5b85c4dc-9tqql"]
Nov 02 22:27:20.680412 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:20.680376    4528 generic.go:356] "Generic (PLEG): container finished" podID="7f71b322-9fb8-4f6c-8662-48c5fb7ee383" containerID="571fd54198e2e9420029c7103d5c1d20e69a4b0ef3353e5900b1cfa856e5cc31" exitCode=0
Nov 02 22:27:20.680797 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:20.680466    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t" event={"ID":"7f71b322-9fb8-4f6c-8662-48c5fb7ee383","Type":"ContainerDied","Data":"571fd54198e2e9420029c7103d5c1d20e69a4b0ef3353e5900b1cfa856e5cc31"}
Nov 02 22:27:20.913127 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:20.913086    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11674de5-804a-4164-8838-f06115b70e7f" path="/var/lib/kubelet/pods/11674de5-804a-4164-8838-f06115b70e7f/volumes"
Nov 02 22:27:21.871644 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:21.871619    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:21.983323 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:21.983237    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-xks-ca\") pod \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") "
Nov 02 22:27:21.983323 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:21.983300    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-kube-api-access-s7wnq\" (UniqueName: \"kubernetes.io/projected/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-kube-api-access-s7wnq\") pod \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") "
Nov 02 22:27:21.983562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:21.983483    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-openshift-ca\") pod \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\" (UID: \"7f71b322-9fb8-4f6c-8662-48c5fb7ee383\") "
Nov 02 22:27:21.983869 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:21.983823    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-openshift-ca" pod "7f71b322-9fb8-4f6c-8662-48c5fb7ee383" (UID: "7f71b322-9fb8-4f6c-8662-48c5fb7ee383"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:21.984010 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:21.983902    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:21.985569 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:21.985534    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-xks-ca" pod "7f71b322-9fb8-4f6c-8662-48c5fb7ee383" (UID: "7f71b322-9fb8-4f6c-8662-48c5fb7ee383"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:27:21.986088 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:21.986062    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-kube-api-access-s7wnq" pod "7f71b322-9fb8-4f6c-8662-48c5fb7ee383" (UID: "7f71b322-9fb8-4f6c-8662-48c5fb7ee383"). InnerVolumeSpecName "kube-api-access-s7wnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:22.085076 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:22.085038    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.085076 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:22.085070    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wnq\" (UniqueName: \"kubernetes.io/projected/7f71b322-9fb8-4f6c-8662-48c5fb7ee383-kube-api-access-s7wnq\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:22.695048 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:22.695018    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"
Nov 02 22:27:22.695250 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:22.695057    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t" event={"ID":"7f71b322-9fb8-4f6c-8662-48c5fb7ee383","Type":"ContainerDied","Data":"e05094c017aefb3b0b9bedfd62c48f5497de1e641d9fcac74f152cc0ceaec6e8"}
Nov 02 22:27:22.695250 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:22.695131    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05094c017aefb3b0b9bedfd62c48f5497de1e641d9fcac74f152cc0ceaec6e8"
Nov 02 22:27:24.442946 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:24.442902    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk" podUID="e7e4a60f-9bee-49b9-b5ac-57598df1aa99" containerName="payload-processing" containerID="cri-o://1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6" gracePeriod=25
Nov 02 22:27:24.539637 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:24.539585    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk" podUID="5c199224-f5a0-4200-8ed3-f807671f0416" containerName="payload-pre-processing" containerID="cri-o://426c2d461e5c17253066751d72020e3a62986c8287dd8ec68d60824618d876a9" gracePeriod=25
Nov 02 22:27:29.627904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.627869    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"]
Nov 02 22:27:29.628455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.628439    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7f71b322-9fb8-4f6c-8662-48c5fb7ee383" containerName="revoke-keys"
Nov 02 22:27:29.628455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.628455    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f71b322-9fb8-4f6c-8662-48c5fb7ee383" containerName="revoke-keys"
Nov 02 22:27:29.628558 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.628550    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ce6aaa-a760-4e75-963b-9c8cfc41fc2a" containerName="istio-proxy"
Nov 02 22:27:29.628595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.628565    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b183f95a-965b-4052-b80f-f7fb5604a50c" containerName="istio-proxy"
Nov 02 22:27:29.628595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.628573    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f71b322-9fb8-4f6c-8662-48c5fb7ee383" containerName="revoke-keys"
Nov 02 22:27:29.632326 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.632305    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.670122 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.670077    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e8cbb8-s5fbz\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.670252 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.670210    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e8cbb8-s5fbz\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.670320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.670282    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjfx\" (UniqueName: \"kubernetes.io/projected/ab850fbc-874a-412f-9685-8e2e6487de13-kube-api-access-pvjfx\") pod \"maas-api-revoke-sub-51e8cbb8-s5fbz\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.688648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.688621    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"]
Nov 02 22:27:29.735225 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.735201    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:27:29.735365 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.735228    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:27:29.735365 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.735266    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:27:29.735454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.735374    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:27:29.771938 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.771907    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e8cbb8-s5fbz\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.772085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.771969    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e8cbb8-s5fbz\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.772085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.772007    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjfx\" (UniqueName: \"kubernetes.io/projected/ab850fbc-874a-412f-9685-8e2e6487de13-kube-api-access-pvjfx\") pod \"maas-api-revoke-sub-51e8cbb8-s5fbz\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.772734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.772709    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e8cbb8-s5fbz\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.774368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.774347    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e8cbb8-s5fbz\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.828813 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.828775    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjfx\" (UniqueName: \"kubernetes.io/projected/ab850fbc-874a-412f-9685-8e2e6487de13-kube-api-access-pvjfx\") pod \"maas-api-revoke-sub-51e8cbb8-s5fbz\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:29.945325 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:29.945243    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:30.316056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:30.315807    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk" podUID="e7e4a60f-9bee-49b9-b5ac-57598df1aa99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:27:30.316056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:30.315887    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk" podUID="5c199224-f5a0-4200-8ed3-f807671f0416" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:27:30.319331 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:27:30.319305    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab850fbc_874a_412f_9685_8e2e6487de13.slice/crio-8854e01352d0e3caad8fd8aa5a26c56fdef232ff9302f28d738714499709df8e WatchSource:0}: Error finding container 8854e01352d0e3caad8fd8aa5a26c56fdef232ff9302f28d738714499709df8e: Status 404 returned error can't find the container with id 8854e01352d0e3caad8fd8aa5a26c56fdef232ff9302f28d738714499709df8e
Nov 02 22:27:30.320166 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:30.320098    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"]
Nov 02 22:27:30.756104 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:30.756067    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz" event={"ID":"ab850fbc-874a-412f-9685-8e2e6487de13","Type":"ContainerStarted","Data":"494a51490c8727a077a51c31536e3068f45f292e755c8c444d955caaa19357a1"}
Nov 02 22:27:30.756583 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:30.756158    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz" event={"ID":"ab850fbc-874a-412f-9685-8e2e6487de13","Type":"ContainerStarted","Data":"8854e01352d0e3caad8fd8aa5a26c56fdef232ff9302f28d738714499709df8e"}
Nov 02 22:27:30.871753 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:30.871650    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz" podStartSLOduration=1.871636149 podStartE2EDuration="1.871636149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:30.870504925 +0000 UTC m=+4738.134041268" watchObservedRunningTime="2026-11-02 22:27:30.871636149 +0000 UTC m=+4738.135172465"
Nov 02 22:27:31.763979 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:31.763944    4528 generic.go:356] "Generic (PLEG): container finished" podID="ab850fbc-874a-412f-9685-8e2e6487de13" containerID="494a51490c8727a077a51c31536e3068f45f292e755c8c444d955caaa19357a1" exitCode=0
Nov 02 22:27:31.764398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:31.763990    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz" event={"ID":"ab850fbc-874a-412f-9685-8e2e6487de13","Type":"ContainerDied","Data":"494a51490c8727a077a51c31536e3068f45f292e755c8c444d955caaa19357a1"}
Nov 02 22:27:32.910316 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:32.910291    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:33.006303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.006275    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-xks-ca\") pod \"ab850fbc-874a-412f-9685-8e2e6487de13\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") "
Nov 02 22:27:33.006477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.006409    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-openshift-ca\") pod \"ab850fbc-874a-412f-9685-8e2e6487de13\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") "
Nov 02 22:27:33.006477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.006468    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab850fbc-874a-412f-9685-8e2e6487de13-kube-api-access-pvjfx\" (UniqueName: \"kubernetes.io/projected/ab850fbc-874a-412f-9685-8e2e6487de13-kube-api-access-pvjfx\") pod \"ab850fbc-874a-412f-9685-8e2e6487de13\" (UID: \"ab850fbc-874a-412f-9685-8e2e6487de13\") "
Nov 02 22:27:33.006742 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.006714    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-openshift-ca" pod "ab850fbc-874a-412f-9685-8e2e6487de13" (UID: "ab850fbc-874a-412f-9685-8e2e6487de13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:33.006884 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.006861    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:33.008842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.008817    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-xks-ca" pod "ab850fbc-874a-412f-9685-8e2e6487de13" (UID: "ab850fbc-874a-412f-9685-8e2e6487de13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:27:33.008939 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.008819    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab850fbc-874a-412f-9685-8e2e6487de13-kube-api-access-pvjfx" pod "ab850fbc-874a-412f-9685-8e2e6487de13" (UID: "ab850fbc-874a-412f-9685-8e2e6487de13"). InnerVolumeSpecName "kube-api-access-pvjfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:33.108388 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.108320    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab850fbc-874a-412f-9685-8e2e6487de13-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:33.108388 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.108346    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjfx\" (UniqueName: \"kubernetes.io/projected/ab850fbc-874a-412f-9685-8e2e6487de13-kube-api-access-pvjfx\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:33.777488 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.777462    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"
Nov 02 22:27:33.777666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.777457    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz" event={"ID":"ab850fbc-874a-412f-9685-8e2e6487de13","Type":"ContainerDied","Data":"8854e01352d0e3caad8fd8aa5a26c56fdef232ff9302f28d738714499709df8e"}
Nov 02 22:27:33.777666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:33.777598    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8854e01352d0e3caad8fd8aa5a26c56fdef232ff9302f28d738714499709df8e"
Nov 02 22:27:36.554156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:36.546033    4528 scope.go:122] "RemoveContainer" containerID="64a966c7a7a66d953695a61e01518e82d0c7238bd9d6287471c25f62a13487fb"
Nov 02 22:27:36.571934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:36.571900    4528 scope.go:122] "RemoveContainer" containerID="b378d96dafe50ca38b99feec624f672ac418f9bc106e4d35988b80cb8d991542"
Nov 02 22:27:40.315971 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:40.315925    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk" podUID="e7e4a60f-9bee-49b9-b5ac-57598df1aa99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:27:40.315971 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:40.315956    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk" podUID="5c199224-f5a0-4200-8ed3-f807671f0416" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:27:44.114354 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:27:44.114317    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:27:44.723667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.723640    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 22:27:44.876427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.876394    4528 generic.go:356] "Generic (PLEG): container finished" podID="e7e4a60f-9bee-49b9-b5ac-57598df1aa99" containerID="1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6" exitCode=0
Nov 02 22:27:44.876633 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.876491    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk" event={"ID":"e7e4a60f-9bee-49b9-b5ac-57598df1aa99","Type":"ContainerDied","Data":"1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6"}
Nov 02 22:27:44.876633 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.876540    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk" event={"ID":"e7e4a60f-9bee-49b9-b5ac-57598df1aa99","Type":"ContainerDied","Data":"fcbb339915be35870d65e009ef32810b5419501a4f45b87f8ba1fd21b543e043"}
Nov 02 22:27:44.876633 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.876570    4528 scope.go:122] "RemoveContainer" containerID="1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6"
Nov 02 22:27:44.876838 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.876820    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"
Nov 02 22:27:44.879900 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.879881    4528 generic.go:356] "Generic (PLEG): container finished" podID="5c199224-f5a0-4200-8ed3-f807671f0416" containerID="426c2d461e5c17253066751d72020e3a62986c8287dd8ec68d60824618d876a9" exitCode=0
Nov 02 22:27:44.880035 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.880023    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk" event={"ID":"5c199224-f5a0-4200-8ed3-f807671f0416","Type":"ContainerDied","Data":"426c2d461e5c17253066751d72020e3a62986c8287dd8ec68d60824618d876a9"}
Nov 02 22:27:44.882349 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.882330    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-kube-api-access-dx4bz\" (UniqueName: \"kubernetes.io/projected/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-kube-api-access-dx4bz\") pod \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\" (UID: \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\") "
Nov 02 22:27:44.882576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.882565    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-plugins-config-volume\") pod \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\" (UID: \"e7e4a60f-9bee-49b9-b5ac-57598df1aa99\") "
Nov 02 22:27:44.883254 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.883235    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-plugins-config-volume" pod "e7e4a60f-9bee-49b9-b5ac-57598df1aa99" (UID: "e7e4a60f-9bee-49b9-b5ac-57598df1aa99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:44.890531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.890206    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-kube-api-access-dx4bz" pod "e7e4a60f-9bee-49b9-b5ac-57598df1aa99" (UID: "e7e4a60f-9bee-49b9-b5ac-57598df1aa99"). InnerVolumeSpecName "kube-api-access-dx4bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:44.898072 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.898038    4528 scope.go:122] "RemoveContainer" containerID="1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6"
Nov 02 22:27:44.898450 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:27:44.898370    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6\": container with ID starting with 1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6 not found: ID does not exist" containerID="1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6"
Nov 02 22:27:44.898450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.898406    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6"} err="failed to get container status \"1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6\": rpc error: code = NotFound desc = could not find container \"1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6\": container with ID starting with 1296df41b09a2e16b74b2ee3925d841b01f36dac94bcfa09b7a5962435b862a6 not found: ID does not exist"
Nov 02 22:27:44.900947 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.900929    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 22:27:44.983876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.983795    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c199224-f5a0-4200-8ed3-f807671f0416-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c199224-f5a0-4200-8ed3-f807671f0416-plugins-config-volume\") pod \"5c199224-f5a0-4200-8ed3-f807671f0416\" (UID: \"5c199224-f5a0-4200-8ed3-f807671f0416\") "
Nov 02 22:27:44.983876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.983873    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c199224-f5a0-4200-8ed3-f807671f0416-kube-api-access-6rnm7\" (UniqueName: \"kubernetes.io/projected/5c199224-f5a0-4200-8ed3-f807671f0416-kube-api-access-6rnm7\") pod \"5c199224-f5a0-4200-8ed3-f807671f0416\" (UID: \"5c199224-f5a0-4200-8ed3-f807671f0416\") "
Nov 02 22:27:44.984249 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.984221    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c199224-f5a0-4200-8ed3-f807671f0416-plugins-config-volume" pod "5c199224-f5a0-4200-8ed3-f807671f0416" (UID: "5c199224-f5a0-4200-8ed3-f807671f0416"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:27:44.984305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.984241    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx4bz\" (UniqueName: \"kubernetes.io/projected/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-kube-api-access-dx4bz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:44.984305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.984275    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4a60f-9bee-49b9-b5ac-57598df1aa99-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:44.985955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:44.985939    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c199224-f5a0-4200-8ed3-f807671f0416-kube-api-access-6rnm7" pod "5c199224-f5a0-4200-8ed3-f807671f0416" (UID: "5c199224-f5a0-4200-8ed3-f807671f0416"). InnerVolumeSpecName "kube-api-access-6rnm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:27:45.100167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:45.097579    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c199224-f5a0-4200-8ed3-f807671f0416-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:45.100167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:45.097616    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnm7\" (UniqueName: \"kubernetes.io/projected/5c199224-f5a0-4200-8ed3-f807671f0416-kube-api-access-6rnm7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:27:45.305390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:45.304703    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"]
Nov 02 22:27:45.308298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:45.308269    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06e461-588c4bf8f7-hpkrk"]
Nov 02 22:27:45.889521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:45.889489    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"
Nov 02 22:27:45.889521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:45.889508    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk" event={"ID":"5c199224-f5a0-4200-8ed3-f807671f0416","Type":"ContainerDied","Data":"1fd59d8aca1717fa6b87b8fd5a84fdeda4182b32497c38e42be617bc72af8b3f"}
Nov 02 22:27:45.889753 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:45.889614    4528 scope.go:122] "RemoveContainer" containerID="426c2d461e5c17253066751d72020e3a62986c8287dd8ec68d60824618d876a9"
Nov 02 22:27:46.056180 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:46.056096    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"]
Nov 02 22:27:46.106156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:46.106120    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06e461-8557d8ff-j7cwk"]
Nov 02 22:27:46.913621 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:46.913587    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c199224-f5a0-4200-8ed3-f807671f0416" path="/var/lib/kubelet/pods/5c199224-f5a0-4200-8ed3-f807671f0416/volumes"
Nov 02 22:27:46.914066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:46.914051    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e4a60f-9bee-49b9-b5ac-57598df1aa99" path="/var/lib/kubelet/pods/e7e4a60f-9bee-49b9-b5ac-57598df1aa99/volumes"
Nov 02 22:27:48.200666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:48.200635    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"]
Nov 02 22:27:48.306514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:48.306482    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"]
Nov 02 22:27:53.201304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:53.201266    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c" podUID="0ad99e9a-8455-4df2-8489-07e0c393d68b" containerName="payload-processing" containerID="cri-o://22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56" gracePeriod=25
Nov 02 22:27:53.307187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:53.307137    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4" podUID="3a493258-30e3-4d3b-beb0-b91417fd4152" containerName="payload-pre-processing" containerID="cri-o://10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53" gracePeriod=25
Nov 02 22:27:53.563945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:53.563908    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4" podUID="3a493258-30e3-4d3b-beb0-b91417fd4152" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:27:53.564152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:53.563963    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c" podUID="0ad99e9a-8455-4df2-8489-07e0c393d68b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:27:59.003140 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:27:59.003068    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"]
Nov 02 22:28:03.563623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:03.563581    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4" podUID="3a493258-30e3-4d3b-beb0-b91417fd4152" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:28:03.564146 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:03.563581    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c" podUID="0ad99e9a-8455-4df2-8489-07e0c393d68b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:28:04.004000 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:04.003901    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj" podUID="6d832b07-2793-4c03-a52b-07f295cf3821" containerName="payload-pre-processing" containerID="cri-o://e90174d26661a9d9637f71fbfbbc074a2505be71986a3906c82f7d1de6e6152a" gracePeriod=25
Nov 02 22:28:10.520182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:10.520144    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj" podUID="6d832b07-2793-4c03-a52b-07f295cf3821" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:28:13.480438 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.480411    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 22:28:13.575865 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.575831    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 22:28:13.636947 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.636914    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad99e9a-8455-4df2-8489-07e0c393d68b-kube-api-access-pdpvk\" (UniqueName: \"kubernetes.io/projected/0ad99e9a-8455-4df2-8489-07e0c393d68b-kube-api-access-pdpvk\") pod \"0ad99e9a-8455-4df2-8489-07e0c393d68b\" (UID: \"0ad99e9a-8455-4df2-8489-07e0c393d68b\") "
Nov 02 22:28:13.637143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.636965    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad99e9a-8455-4df2-8489-07e0c393d68b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad99e9a-8455-4df2-8489-07e0c393d68b-plugins-config-volume\") pod \"0ad99e9a-8455-4df2-8489-07e0c393d68b\" (UID: \"0ad99e9a-8455-4df2-8489-07e0c393d68b\") "
Nov 02 22:28:13.637386 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.637364    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad99e9a-8455-4df2-8489-07e0c393d68b-plugins-config-volume" pod "0ad99e9a-8455-4df2-8489-07e0c393d68b" (UID: "0ad99e9a-8455-4df2-8489-07e0c393d68b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:28:13.638980 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.638960    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad99e9a-8455-4df2-8489-07e0c393d68b-kube-api-access-pdpvk" pod "0ad99e9a-8455-4df2-8489-07e0c393d68b" (UID: "0ad99e9a-8455-4df2-8489-07e0c393d68b"). InnerVolumeSpecName "kube-api-access-pdpvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:28:13.738297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.738208    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a493258-30e3-4d3b-beb0-b91417fd4152-kube-api-access-664sl\" (UniqueName: \"kubernetes.io/projected/3a493258-30e3-4d3b-beb0-b91417fd4152-kube-api-access-664sl\") pod \"3a493258-30e3-4d3b-beb0-b91417fd4152\" (UID: \"3a493258-30e3-4d3b-beb0-b91417fd4152\") "
Nov 02 22:28:13.738470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.738337    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a493258-30e3-4d3b-beb0-b91417fd4152-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a493258-30e3-4d3b-beb0-b91417fd4152-plugins-config-volume\") pod \"3a493258-30e3-4d3b-beb0-b91417fd4152\" (UID: \"3a493258-30e3-4d3b-beb0-b91417fd4152\") "
Nov 02 22:28:13.738696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.738673    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a493258-30e3-4d3b-beb0-b91417fd4152-plugins-config-volume" pod "3a493258-30e3-4d3b-beb0-b91417fd4152" (UID: "3a493258-30e3-4d3b-beb0-b91417fd4152"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:28:13.738836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.738820    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdpvk\" (UniqueName: \"kubernetes.io/projected/0ad99e9a-8455-4df2-8489-07e0c393d68b-kube-api-access-pdpvk\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:13.738904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.738837    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad99e9a-8455-4df2-8489-07e0c393d68b-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:13.738904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.738847    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a493258-30e3-4d3b-beb0-b91417fd4152-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:13.740411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.740381    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a493258-30e3-4d3b-beb0-b91417fd4152-kube-api-access-664sl" pod "3a493258-30e3-4d3b-beb0-b91417fd4152" (UID: "3a493258-30e3-4d3b-beb0-b91417fd4152"). InnerVolumeSpecName "kube-api-access-664sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:28:13.840136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:13.840094    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-664sl\" (UniqueName: \"kubernetes.io/projected/3a493258-30e3-4d3b-beb0-b91417fd4152-kube-api-access-664sl\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:14.069885 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.069852    4528 generic.go:356] "Generic (PLEG): container finished" podID="0ad99e9a-8455-4df2-8489-07e0c393d68b" containerID="22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56" exitCode=0
Nov 02 22:28:14.070095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.069893    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"
Nov 02 22:28:14.070095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.069939    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c" event={"ID":"0ad99e9a-8455-4df2-8489-07e0c393d68b","Type":"ContainerDied","Data":"22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56"}
Nov 02 22:28:14.070095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.069986    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c" event={"ID":"0ad99e9a-8455-4df2-8489-07e0c393d68b","Type":"ContainerDied","Data":"aeb2db92e6048aeb96df66ee3e937b43ce4c79bcdec2401f62dcdd3800c2d1c7"}
Nov 02 22:28:14.070095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.070012    4528 scope.go:122] "RemoveContainer" containerID="22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56"
Nov 02 22:28:14.071486 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.071462    4528 generic.go:356] "Generic (PLEG): container finished" podID="3a493258-30e3-4d3b-beb0-b91417fd4152" containerID="10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53" exitCode=0
Nov 02 22:28:14.071604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.071522    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"
Nov 02 22:28:14.071604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.071563    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4" event={"ID":"3a493258-30e3-4d3b-beb0-b91417fd4152","Type":"ContainerDied","Data":"10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53"}
Nov 02 22:28:14.071604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.071598    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4" event={"ID":"3a493258-30e3-4d3b-beb0-b91417fd4152","Type":"ContainerDied","Data":"27cfab9d25f43e0d58e00710e0e53486b4f37c807793ee96eb2503a9d3450338"}
Nov 02 22:28:14.082095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.082080    4528 scope.go:122] "RemoveContainer" containerID="22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56"
Nov 02 22:28:14.082428 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:28:14.082403    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56\": container with ID starting with 22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56 not found: ID does not exist" containerID="22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56"
Nov 02 22:28:14.082516 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.082436    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56"} err="failed to get container status \"22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56\": rpc error: code = NotFound desc = could not find container \"22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56\": container with ID starting with 22a6268bb44799e48087d4046f660b50aeb9070502d7c8ddee729f51d8fa0f56 not found: ID does not exist"
Nov 02 22:28:14.082516 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.082451    4528 scope.go:122] "RemoveContainer" containerID="10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53"
Nov 02 22:28:14.091036 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.091020    4528 scope.go:122] "RemoveContainer" containerID="10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53"
Nov 02 22:28:14.091318 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:28:14.091301    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53\": container with ID starting with 10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53 not found: ID does not exist" containerID="10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53"
Nov 02 22:28:14.091375 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.091323    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53"} err="failed to get container status \"10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53\": rpc error: code = NotFound desc = could not find container \"10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53\": container with ID starting with 10ca98c6e9bcf2cf50a758dad3897aa919701160e1f0f3be9ba6f5784de54c53 not found: ID does not exist"
Nov 02 22:28:14.112228 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:28:14.112202    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:28:14.236271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.236244    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"]
Nov 02 22:28:14.239762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.239738    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4"]
Nov 02 22:28:14.349330 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.349254    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"]
Nov 02 22:28:14.357732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.357703    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c7e71f-578bc48b4f-bpz2c"]
Nov 02 22:28:14.563500 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.563461    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c7e71f-7bcf5dbf6c-j8bn4" podUID="3a493258-30e3-4d3b-beb0-b91417fd4152" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.4:50052\" within 1s: context deadline exceeded"
Nov 02 22:28:14.563910 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:28:14.563519    4528 logging.go:55] [core] [Channel #20794 SubChannel #20795]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.4:50052", ServerName: "10.133.1.4:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.4:50052: operation was canceled"
Nov 02 22:28:14.913598 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.913567    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad99e9a-8455-4df2-8489-07e0c393d68b" path="/var/lib/kubelet/pods/0ad99e9a-8455-4df2-8489-07e0c393d68b/volumes"
Nov 02 22:28:14.913978 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:14.913965    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a493258-30e3-4d3b-beb0-b91417fd4152" path="/var/lib/kubelet/pods/3a493258-30e3-4d3b-beb0-b91417fd4152/volumes"
Nov 02 22:28:17.768466 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:17.768428    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:28:20.519553 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:20.519517    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj" podUID="6d832b07-2793-4c03-a52b-07f295cf3821" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:28:24.031218 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.031188    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"]
Nov 02 22:28:24.031615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.031411    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" containerID="cri-o://deee69be57f224996c7d7dfd844aa04d97713cb3a76301fe6b6d5b56fb9df6f4" gracePeriod=30
Nov 02 22:28:24.142647 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.142518    4528 generic.go:356] "Generic (PLEG): container finished" podID="6d832b07-2793-4c03-a52b-07f295cf3821" containerID="e90174d26661a9d9637f71fbfbbc074a2505be71986a3906c82f7d1de6e6152a" exitCode=0
Nov 02 22:28:24.142878 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.142653    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj" event={"ID":"6d832b07-2793-4c03-a52b-07f295cf3821","Type":"ContainerDied","Data":"e90174d26661a9d9637f71fbfbbc074a2505be71986a3906c82f7d1de6e6152a"}
Nov 02 22:28:24.275504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.275480    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:28:24.353406 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.353377    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d832b07-2793-4c03-a52b-07f295cf3821-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d832b07-2793-4c03-a52b-07f295cf3821-plugins-config-volume\") pod \"6d832b07-2793-4c03-a52b-07f295cf3821\" (UID: \"6d832b07-2793-4c03-a52b-07f295cf3821\") "
Nov 02 22:28:24.353848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.353819    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d832b07-2793-4c03-a52b-07f295cf3821-plugins-config-volume" pod "6d832b07-2793-4c03-a52b-07f295cf3821" (UID: "6d832b07-2793-4c03-a52b-07f295cf3821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:28:24.353944 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.353883    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d832b07-2793-4c03-a52b-07f295cf3821-kube-api-access-bc6cc\" (UniqueName: \"kubernetes.io/projected/6d832b07-2793-4c03-a52b-07f295cf3821-kube-api-access-bc6cc\") pod \"6d832b07-2793-4c03-a52b-07f295cf3821\" (UID: \"6d832b07-2793-4c03-a52b-07f295cf3821\") "
Nov 02 22:28:24.354437 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.354408    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d832b07-2793-4c03-a52b-07f295cf3821-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:24.356533 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.356507    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d832b07-2793-4c03-a52b-07f295cf3821-kube-api-access-bc6cc" pod "6d832b07-2793-4c03-a52b-07f295cf3821" (UID: "6d832b07-2793-4c03-a52b-07f295cf3821"). InnerVolumeSpecName "kube-api-access-bc6cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:28:24.455708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:24.455677    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc6cc\" (UniqueName: \"kubernetes.io/projected/6d832b07-2793-4c03-a52b-07f295cf3821-kube-api-access-bc6cc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:25.150235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:25.150199    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj" event={"ID":"6d832b07-2793-4c03-a52b-07f295cf3821","Type":"ContainerDied","Data":"6113c56ad28dd3cd04e4940f9aceb68bb6fa02c9388161b5dc36f87cdad8439f"}
Nov 02 22:28:25.150235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:25.150234    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"
Nov 02 22:28:25.150793 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:25.150273    4528 scope.go:122] "RemoveContainer" containerID="e90174d26661a9d9637f71fbfbbc074a2505be71986a3906c82f7d1de6e6152a"
Nov 02 22:28:25.316167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:25.316136    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"]
Nov 02 22:28:25.367798 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:25.366364    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf2a53-6f7b85f575-fk6pj"]
Nov 02 22:28:26.917157 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:26.917124    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d832b07-2793-4c03-a52b-07f295cf3821" path="/var/lib/kubelet/pods/6d832b07-2793-4c03-a52b-07f295cf3821/volumes"
Nov 02 22:28:28.448951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:28.448914    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lrnwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:28:28.449334 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:28.448980    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:28:29.177866 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.177836    4528 generic.go:356] "Generic (PLEG): container finished" podID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerID="deee69be57f224996c7d7dfd844aa04d97713cb3a76301fe6b6d5b56fb9df6f4" exitCode=0
Nov 02 22:28:29.178033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.177914    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" event={"ID":"bfe3a4bb-818f-45fd-b27b-688c67d97929","Type":"ContainerDied","Data":"deee69be57f224996c7d7dfd844aa04d97713cb3a76301fe6b6d5b56fb9df6f4"}
Nov 02 22:28:29.290691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.290667    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:28:29.416222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416154    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-envoy\") pod \"bfe3a4bb-818f-45fd-b27b-688c67d97929\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") "
Nov 02 22:28:29.416222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416211    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-socket\") pod \"bfe3a4bb-818f-45fd-b27b-688c67d97929\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") "
Nov 02 22:28:29.416411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416267    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-podinfo\") pod \"bfe3a4bb-818f-45fd-b27b-688c67d97929\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") "
Nov 02 22:28:29.416411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416324    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-credential-socket\") pod \"bfe3a4bb-818f-45fd-b27b-688c67d97929\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") "
Nov 02 22:28:29.416411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416396    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-kube-api-access-2jk4q\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-kube-api-access-2jk4q\") pod \"bfe3a4bb-818f-45fd-b27b-688c67d97929\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") "
Nov 02 22:28:29.416591 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416438    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-socket" pod "bfe3a4bb-818f-45fd-b27b-688c67d97929" (UID: "bfe3a4bb-818f-45fd-b27b-688c67d97929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:28:29.416591 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416504    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-data\") pod \"bfe3a4bb-818f-45fd-b27b-688c67d97929\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") "
Nov 02 22:28:29.416591 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416536    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe3a4bb-818f-45fd-b27b-688c67d97929-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe3a4bb-818f-45fd-b27b-688c67d97929-istiod-ca-cert\") pod \"bfe3a4bb-818f-45fd-b27b-688c67d97929\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") "
Nov 02 22:28:29.416755 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416621    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-token\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-token\") pod \"bfe3a4bb-818f-45fd-b27b-688c67d97929\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") "
Nov 02 22:28:29.416755 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416647    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-certs\") pod \"bfe3a4bb-818f-45fd-b27b-688c67d97929\" (UID: \"bfe3a4bb-818f-45fd-b27b-688c67d97929\") "
Nov 02 22:28:29.416755 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416652    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-credential-socket" pod "bfe3a4bb-818f-45fd-b27b-688c67d97929" (UID: "bfe3a4bb-818f-45fd-b27b-688c67d97929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:28:29.416935 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416764    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-data" pod "bfe3a4bb-818f-45fd-b27b-688c67d97929" (UID: "bfe3a4bb-818f-45fd-b27b-688c67d97929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:28:29.416935 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.416896    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-certs" pod "bfe3a4bb-818f-45fd-b27b-688c67d97929" (UID: "bfe3a4bb-818f-45fd-b27b-688c67d97929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:28:29.417152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.417101    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:29.417296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.417157    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:29.417296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.417173    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:29.417296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.417186    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:29.417296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.417231    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe3a4bb-818f-45fd-b27b-688c67d97929-istiod-ca-cert" pod "bfe3a4bb-818f-45fd-b27b-688c67d97929" (UID: "bfe3a4bb-818f-45fd-b27b-688c67d97929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:28:29.418549 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.418526    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-kube-api-access-2jk4q" pod "bfe3a4bb-818f-45fd-b27b-688c67d97929" (UID: "bfe3a4bb-818f-45fd-b27b-688c67d97929"). InnerVolumeSpecName "kube-api-access-2jk4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:28:29.418639 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.418585    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-token" pod "bfe3a4bb-818f-45fd-b27b-688c67d97929" (UID: "bfe3a4bb-818f-45fd-b27b-688c67d97929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:28:29.418809 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.418789    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-podinfo" pod "bfe3a4bb-818f-45fd-b27b-688c67d97929" (UID: "bfe3a4bb-818f-45fd-b27b-688c67d97929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:28:29.418899 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.418838    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-envoy" pod "bfe3a4bb-818f-45fd-b27b-688c67d97929" (UID: "bfe3a4bb-818f-45fd-b27b-688c67d97929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:28:29.518255 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.518224    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe3a4bb-818f-45fd-b27b-688c67d97929-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:29.518255 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.518249    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:29.518255 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.518259    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:29.518685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.518268    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe3a4bb-818f-45fd-b27b-688c67d97929-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:29.518685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:29.518276    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jk4q\" (UniqueName: \"kubernetes.io/projected/bfe3a4bb-818f-45fd-b27b-688c67d97929-kube-api-access-2jk4q\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:30.185684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:30.185652    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"
Nov 02 22:28:30.185860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:30.185687    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd" event={"ID":"bfe3a4bb-818f-45fd-b27b-688c67d97929","Type":"ContainerDied","Data":"e1c091aae3257a1ed288a30fefb2da3b97ac014c71ff258148be7b02908599cf"}
Nov 02 22:28:30.185860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:30.185764    4528 scope.go:122] "RemoveContainer" containerID="deee69be57f224996c7d7dfd844aa04d97713cb3a76301fe6b6d5b56fb9df6f4"
Nov 02 22:28:30.402013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:30.401971    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"]
Nov 02 22:28:30.404994 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:30.404971    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lrnwd"]
Nov 02 22:28:30.912875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:30.912840    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" path="/var/lib/kubelet/pods/bfe3a4bb-818f-45fd-b27b-688c67d97929/volumes"
Nov 02 22:28:31.988378 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:31.988346    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"]
Nov 02 22:28:36.989598 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:36.989514    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g" podUID="b0f4ec96-a910-44dd-aab7-245f8a0471f4" containerName="payload-pre-processing" containerID="cri-o://2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33" gracePeriod=25
Nov 02 22:28:44.114105 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:28:44.114070    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:28:46.832870 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:46.832828    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g" podUID="b0f4ec96-a910-44dd-aab7-245f8a0471f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:28:56.832661 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:56.832618    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g" podUID="b0f4ec96-a910-44dd-aab7-245f8a0471f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:28:57.262056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.262018    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 22:28:57.341531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.341497    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f4ec96-a910-44dd-aab7-245f8a0471f4-kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/b0f4ec96-a910-44dd-aab7-245f8a0471f4-kube-api-access-qpwks\") pod \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\" (UID: \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\") "
Nov 02 22:28:57.341741 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.341548    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f4ec96-a910-44dd-aab7-245f8a0471f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f4ec96-a910-44dd-aab7-245f8a0471f4-plugins-config-volume\") pod \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\" (UID: \"b0f4ec96-a910-44dd-aab7-245f8a0471f4\") "
Nov 02 22:28:57.342033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.342005    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f4ec96-a910-44dd-aab7-245f8a0471f4-plugins-config-volume" pod "b0f4ec96-a910-44dd-aab7-245f8a0471f4" (UID: "b0f4ec96-a910-44dd-aab7-245f8a0471f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:28:57.352991 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.352898    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f4ec96-a910-44dd-aab7-245f8a0471f4-kube-api-access-qpwks" pod "b0f4ec96-a910-44dd-aab7-245f8a0471f4" (UID: "b0f4ec96-a910-44dd-aab7-245f8a0471f4"). InnerVolumeSpecName "kube-api-access-qpwks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:28:57.363943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.363909    4528 generic.go:356] "Generic (PLEG): container finished" podID="b0f4ec96-a910-44dd-aab7-245f8a0471f4" containerID="2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33" exitCode=0
Nov 02 22:28:57.364088 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.363964    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"
Nov 02 22:28:57.364088 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.363970    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g" event={"ID":"b0f4ec96-a910-44dd-aab7-245f8a0471f4","Type":"ContainerDied","Data":"2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33"}
Nov 02 22:28:57.364088 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.364058    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g" event={"ID":"b0f4ec96-a910-44dd-aab7-245f8a0471f4","Type":"ContainerDied","Data":"0940ad00b8c0a0a021421f060c39125752cd1f50d324dadfa25cc43f9866d6e5"}
Nov 02 22:28:57.364088 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.364087    4528 scope.go:122] "RemoveContainer" containerID="2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33"
Nov 02 22:28:57.375493 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.375474    4528 scope.go:122] "RemoveContainer" containerID="2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33"
Nov 02 22:28:57.375810 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:28:57.375781    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33\": container with ID starting with 2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33 not found: ID does not exist" containerID="2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33"
Nov 02 22:28:57.375923 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.375819    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33"} err="failed to get container status \"2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33\": rpc error: code = NotFound desc = could not find container \"2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33\": container with ID starting with 2d3e7161eeec7c5e38b63bcc0fa054f17206e7612ab830a20bbe2f3795044e33 not found: ID does not exist"
Nov 02 22:28:57.443206 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.443176    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpwks\" (UniqueName: \"kubernetes.io/projected/b0f4ec96-a910-44dd-aab7-245f8a0471f4-kube-api-access-qpwks\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:57.443206 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.443200    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f4ec96-a910-44dd-aab7-245f8a0471f4-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:28:57.486942 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.486910    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"]
Nov 02 22:28:57.489983 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:57.489956    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7cde84-b4c847df9-7bm7g"]
Nov 02 22:28:58.913174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:28:58.913142    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f4ec96-a910-44dd-aab7-245f8a0471f4" path="/var/lib/kubelet/pods/b0f4ec96-a910-44dd-aab7-245f8a0471f4/volumes"
Nov 02 22:29:01.081231 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:01.081200    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"]
Nov 02 22:29:01.081653 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:01.081392    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy" containerID="cri-o://38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0" gracePeriod=30
Nov 02 22:29:02.140426 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.140389    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"]
Nov 02 22:29:02.140975 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.140955    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3a493258-30e3-4d3b-beb0-b91417fd4152" containerName="payload-pre-processing"
Nov 02 22:29:02.141033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.140977    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a493258-30e3-4d3b-beb0-b91417fd4152" containerName="payload-pre-processing"
Nov 02 22:29:02.141033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141014    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e7e4a60f-9bee-49b9-b5ac-57598df1aa99" containerName="payload-processing"
Nov 02 22:29:02.141033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141023    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e4a60f-9bee-49b9-b5ac-57598df1aa99" containerName="payload-processing"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141044    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6d832b07-2793-4c03-a52b-07f295cf3821" containerName="payload-pre-processing"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141051    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d832b07-2793-4c03-a52b-07f295cf3821" containerName="payload-pre-processing"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141072    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ab850fbc-874a-412f-9685-8e2e6487de13" containerName="revoke-keys"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141081    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab850fbc-874a-412f-9685-8e2e6487de13" containerName="revoke-keys"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141096    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141104    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141138    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5c199224-f5a0-4200-8ed3-f807671f0416" containerName="payload-pre-processing"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141146    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c199224-f5a0-4200-8ed3-f807671f0416" containerName="payload-pre-processing"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141160    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b0f4ec96-a910-44dd-aab7-245f8a0471f4" containerName="payload-pre-processing"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141168    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f4ec96-a910-44dd-aab7-245f8a0471f4" containerName="payload-pre-processing"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141185    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0ad99e9a-8455-4df2-8489-07e0c393d68b" containerName="payload-processing"
Nov 02 22:29:02.141191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141191    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad99e9a-8455-4df2-8489-07e0c393d68b" containerName="payload-processing"
Nov 02 22:29:02.141608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141319    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab850fbc-874a-412f-9685-8e2e6487de13" containerName="revoke-keys"
Nov 02 22:29:02.141608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141333    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe3a4bb-818f-45fd-b27b-688c67d97929" containerName="istio-proxy"
Nov 02 22:29:02.141608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141343    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c199224-f5a0-4200-8ed3-f807671f0416" containerName="payload-pre-processing"
Nov 02 22:29:02.141608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141353    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f4ec96-a910-44dd-aab7-245f8a0471f4" containerName="payload-pre-processing"
Nov 02 22:29:02.141608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141362    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad99e9a-8455-4df2-8489-07e0c393d68b" containerName="payload-processing"
Nov 02 22:29:02.141608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141371    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a493258-30e3-4d3b-beb0-b91417fd4152" containerName="payload-pre-processing"
Nov 02 22:29:02.141608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141384    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e4a60f-9bee-49b9-b5ac-57598df1aa99" containerName="payload-processing"
Nov 02 22:29:02.141608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.141394    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d832b07-2793-4c03-a52b-07f295cf3821" containerName="payload-pre-processing"
Nov 02 22:29:02.146422 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.146405    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.201481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.201449    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"]
Nov 02 22:29:02.204212 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.204182    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxgh\" (UniqueName: \"kubernetes.io/projected/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-kube-api-access-ctxgh\") pod \"maas-api-revoke-sub-7ae51d07-8vtkh\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.204357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.204259    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae51d07-8vtkh\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.204417 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.204364    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae51d07-8vtkh\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.248372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.248330    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:29:02.248372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.248348    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:29:02.248372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.248356    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:29:02.249302 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.249288    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:29:02.305904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.305871    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae51d07-8vtkh\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.306079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.306027    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxgh\" (UniqueName: \"kubernetes.io/projected/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-kube-api-access-ctxgh\") pod \"maas-api-revoke-sub-7ae51d07-8vtkh\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.306079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.306067    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae51d07-8vtkh\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.306702 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.306677    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae51d07-8vtkh\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.308228 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.308207    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae51d07-8vtkh\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.364884 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.364850    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxgh\" (UniqueName: \"kubernetes.io/projected/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-kube-api-access-ctxgh\") pod \"maas-api-revoke-sub-7ae51d07-8vtkh\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.459504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.459437    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:02.893941 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:29:02.893871    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd907fbc4_2060_47ed_81f8_2067ee3f5a1c.slice/crio-22fef2896e19775afcbc9224cd56a23938148d5f0832b8831e5c510466436d82 WatchSource:0}: Error finding container 22fef2896e19775afcbc9224cd56a23938148d5f0832b8831e5c510466436d82: Status 404 returned error can't find the container with id 22fef2896e19775afcbc9224cd56a23938148d5f0832b8831e5c510466436d82
Nov 02 22:29:02.896922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:02.896625    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"]
Nov 02 22:29:03.412599 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.412549    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh" event={"ID":"d907fbc4-2060-47ed-81f8-2067ee3f5a1c","Type":"ContainerStarted","Data":"29d318222050759608512a6740aa89c98286aed87eb6cfefa4cae5e73f7b7633"}
Nov 02 22:29:03.413011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.412649    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh" event={"ID":"d907fbc4-2060-47ed-81f8-2067ee3f5a1c","Type":"ContainerStarted","Data":"22fef2896e19775afcbc9224cd56a23938148d5f0832b8831e5c510466436d82"}
Nov 02 22:29:03.487329 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.487257    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"]
Nov 02 22:29:03.491138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.491105    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.529380 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.529332    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh" podStartSLOduration=1.529319308 podStartE2EDuration="1.529319308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:03.527236365 +0000 UTC m=+4830.790772683" watchObservedRunningTime="2026-11-02 22:29:03.529319308 +0000 UTC m=+4830.792855625"
Nov 02 22:29:03.547444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.547417    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"]
Nov 02 22:29:03.626665 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.626630    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvr2r\" (UniqueName: \"kubernetes.io/projected/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-kube-api-access-gvr2r\") pod \"maas-api-revoke-sub-b1baa6e4-d8pf4\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.626826 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.626754    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1baa6e4-d8pf4\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.626870 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.626830    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1baa6e4-d8pf4\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.727962 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.727934    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvr2r\" (UniqueName: \"kubernetes.io/projected/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-kube-api-access-gvr2r\") pod \"maas-api-revoke-sub-b1baa6e4-d8pf4\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.728175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.728015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1baa6e4-d8pf4\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.728175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.728072    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1baa6e4-d8pf4\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.728718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.728695    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1baa6e4-d8pf4\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.730395 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.730367    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1baa6e4-d8pf4\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.784544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.784520    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvr2r\" (UniqueName: \"kubernetes.io/projected/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-kube-api-access-gvr2r\") pod \"maas-api-revoke-sub-b1baa6e4-d8pf4\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:03.805248 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:03.805225    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:04.177202 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:29:04.177168    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ba4b13_99a3_4d64_9a15_5907bd0ef7cc.slice/crio-3131b9353a64134af0c022b908e2a46b98c252741ace61a8e18d5676020d1536 WatchSource:0}: Error finding container 3131b9353a64134af0c022b908e2a46b98c252741ace61a8e18d5676020d1536: Status 404 returned error can't find the container with id 3131b9353a64134af0c022b908e2a46b98c252741ace61a8e18d5676020d1536
Nov 02 22:29:04.179746 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:04.179718    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"]
Nov 02 22:29:04.419175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:04.419132    4528 generic.go:356] "Generic (PLEG): container finished" podID="d907fbc4-2060-47ed-81f8-2067ee3f5a1c" containerID="29d318222050759608512a6740aa89c98286aed87eb6cfefa4cae5e73f7b7633" exitCode=0
Nov 02 22:29:04.419625 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:04.419204    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh" event={"ID":"d907fbc4-2060-47ed-81f8-2067ee3f5a1c","Type":"ContainerDied","Data":"29d318222050759608512a6740aa89c98286aed87eb6cfefa4cae5e73f7b7633"}
Nov 02 22:29:04.420490 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:04.420464    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4" event={"ID":"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc","Type":"ContainerStarted","Data":"3131b9353a64134af0c022b908e2a46b98c252741ace61a8e18d5676020d1536"}
Nov 02 22:29:05.428030 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.428001    4528 generic.go:356] "Generic (PLEG): container finished" podID="f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc" containerID="51ee0317de288ecd04fd411f610161a7f2b401c102e8e7d9c21e41a69417496d" exitCode=0
Nov 02 22:29:05.428492 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.428065    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4" event={"ID":"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc","Type":"ContainerDied","Data":"51ee0317de288ecd04fd411f610161a7f2b401c102e8e7d9c21e41a69417496d"}
Nov 02 22:29:05.571511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.571484    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:05.649650 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.649619    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-openshift-ca\") pod \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") "
Nov 02 22:29:05.649845 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.649695    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-xks-ca\") pod \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") "
Nov 02 22:29:05.649845 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.649791    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-kube-api-access-ctxgh\" (UniqueName: \"kubernetes.io/projected/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-kube-api-access-ctxgh\") pod \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\" (UID: \"d907fbc4-2060-47ed-81f8-2067ee3f5a1c\") "
Nov 02 22:29:05.650045 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.650021    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-openshift-ca" pod "d907fbc4-2060-47ed-81f8-2067ee3f5a1c" (UID: "d907fbc4-2060-47ed-81f8-2067ee3f5a1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:05.650235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.650218    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:05.651777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.651755    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-kube-api-access-ctxgh" pod "d907fbc4-2060-47ed-81f8-2067ee3f5a1c" (UID: "d907fbc4-2060-47ed-81f8-2067ee3f5a1c"). InnerVolumeSpecName "kube-api-access-ctxgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:05.651889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.651838    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-xks-ca" pod "d907fbc4-2060-47ed-81f8-2067ee3f5a1c" (UID: "d907fbc4-2060-47ed-81f8-2067ee3f5a1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:05.751595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.751508    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:05.751595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:05.751539    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxgh\" (UniqueName: \"kubernetes.io/projected/d907fbc4-2060-47ed-81f8-2067ee3f5a1c-kube-api-access-ctxgh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.336014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.335988    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 22:29:06.435057 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.434972    4528 generic.go:356] "Generic (PLEG): container finished" podID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerID="38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0" exitCode=0
Nov 02 22:29:06.435057 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.435028    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"
Nov 02 22:29:06.435565 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.435060    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" event={"ID":"d474a4c1-c212-4544-8e0c-f9e411b5b219","Type":"ContainerDied","Data":"38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0"}
Nov 02 22:29:06.435565 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.435157    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk" event={"ID":"d474a4c1-c212-4544-8e0c-f9e411b5b219","Type":"ContainerDied","Data":"1eff552549c0fb03d62291dcac00d29cf834a57d5f29a9b05f16c2a5d15d12b2"}
Nov 02 22:29:06.435565 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.435192    4528 scope.go:122] "RemoveContainer" containerID="38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0"
Nov 02 22:29:06.436423 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.436401    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"
Nov 02 22:29:06.436482 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.436433    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh" event={"ID":"d907fbc4-2060-47ed-81f8-2067ee3f5a1c","Type":"ContainerDied","Data":"22fef2896e19775afcbc9224cd56a23938148d5f0832b8831e5c510466436d82"}
Nov 02 22:29:06.436482 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.436467    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22fef2896e19775afcbc9224cd56a23938148d5f0832b8831e5c510466436d82"
Nov 02 22:29:06.446524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.446508    4528 scope.go:122] "RemoveContainer" containerID="723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117"
Nov 02 22:29:06.455701 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.455680    4528 scope.go:122] "RemoveContainer" containerID="38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0"
Nov 02 22:29:06.455976 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:06.455957    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0\": container with ID starting with 38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0 not found: ID does not exist" containerID="38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0"
Nov 02 22:29:06.456031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.455985    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0"} err="failed to get container status \"38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0\": rpc error: code = NotFound desc = could not find container \"38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0\": container with ID starting with 38b2bebfb06ba6bc076eef4e834c42aaa3465609db09f250fc4add48898e2ef0 not found: ID does not exist"
Nov 02 22:29:06.456031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.456003    4528 scope.go:122] "RemoveContainer" containerID="723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117"
Nov 02 22:29:06.456244 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:06.456219    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117\": container with ID starting with 723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117 not found: ID does not exist" containerID="723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117"
Nov 02 22:29:06.456244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.456235    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117"} err="failed to get container status \"723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117\": rpc error: code = NotFound desc = could not find container \"723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117\": container with ID starting with 723a075d62a21d749be0004d5127b68a7d7115b833c79c9bc0e2ca4043210117 not found: ID does not exist"
Nov 02 22:29:06.459690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.459669    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-data\") pod \"d474a4c1-c212-4544-8e0c-f9e411b5b219\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") "
Nov 02 22:29:06.459794 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.459780    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-certs\") pod \"d474a4c1-c212-4544-8e0c-f9e411b5b219\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") "
Nov 02 22:29:06.459862 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.459818    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d474a4c1-c212-4544-8e0c-f9e411b5b219-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474a4c1-c212-4544-8e0c-f9e411b5b219-istiod-ca-cert\") pod \"d474a4c1-c212-4544-8e0c-f9e411b5b219\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") "
Nov 02 22:29:06.459926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.459857    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-podinfo\") pod \"d474a4c1-c212-4544-8e0c-f9e411b5b219\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") "
Nov 02 22:29:06.459926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.459896    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-kube-api-access-zpfvf\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-kube-api-access-zpfvf\") pod \"d474a4c1-c212-4544-8e0c-f9e411b5b219\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") "
Nov 02 22:29:06.459926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.459899    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-data" pod "d474a4c1-c212-4544-8e0c-f9e411b5b219" (UID: "d474a4c1-c212-4544-8e0c-f9e411b5b219"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:06.460085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.459933    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-envoy\") pod \"d474a4c1-c212-4544-8e0c-f9e411b5b219\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") "
Nov 02 22:29:06.460085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.459973    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-socket\") pod \"d474a4c1-c212-4544-8e0c-f9e411b5b219\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") "
Nov 02 22:29:06.460085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460037    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-token\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-token\") pod \"d474a4c1-c212-4544-8e0c-f9e411b5b219\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") "
Nov 02 22:29:06.460085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460042    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-certs" pod "d474a4c1-c212-4544-8e0c-f9e411b5b219" (UID: "d474a4c1-c212-4544-8e0c-f9e411b5b219"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:06.460085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460070    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-credential-socket\") pod \"d474a4c1-c212-4544-8e0c-f9e411b5b219\" (UID: \"d474a4c1-c212-4544-8e0c-f9e411b5b219\") "
Nov 02 22:29:06.460358 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460311    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-socket" pod "d474a4c1-c212-4544-8e0c-f9e411b5b219" (UID: "d474a4c1-c212-4544-8e0c-f9e411b5b219"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:06.460358 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460235    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d474a4c1-c212-4544-8e0c-f9e411b5b219-istiod-ca-cert" pod "d474a4c1-c212-4544-8e0c-f9e411b5b219" (UID: "d474a4c1-c212-4544-8e0c-f9e411b5b219"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:06.460578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460520    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-credential-socket" pod "d474a4c1-c212-4544-8e0c-f9e411b5b219" (UID: "d474a4c1-c212-4544-8e0c-f9e411b5b219"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:06.460700 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460685    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.460765 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460705    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.460765 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460720    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d474a4c1-c212-4544-8e0c-f9e411b5b219-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.460765 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460735    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.460765 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.460751    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.462007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.461981    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-podinfo" pod "d474a4c1-c212-4544-8e0c-f9e411b5b219" (UID: "d474a4c1-c212-4544-8e0c-f9e411b5b219"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:29:06.462484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.462457    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-envoy" pod "d474a4c1-c212-4544-8e0c-f9e411b5b219" (UID: "d474a4c1-c212-4544-8e0c-f9e411b5b219"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:06.462572 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.462533    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-kube-api-access-zpfvf" pod "d474a4c1-c212-4544-8e0c-f9e411b5b219" (UID: "d474a4c1-c212-4544-8e0c-f9e411b5b219"). InnerVolumeSpecName "kube-api-access-zpfvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:06.462802 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.462780    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-token" pod "d474a4c1-c212-4544-8e0c-f9e411b5b219" (UID: "d474a4c1-c212-4544-8e0c-f9e411b5b219"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:06.496071 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.496039    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"]
Nov 02 22:29:06.562016 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.561990    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.562016 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.562015    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpfvf\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-kube-api-access-zpfvf\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.562222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.562026    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.562222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.562037    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d474a4c1-c212-4544-8e0c-f9e411b5b219-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.585564 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.585536    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:06.763333 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.763291    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-xks-ca\") pod \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") "
Nov 02 22:29:06.763488 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.763419    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-openshift-ca\") pod \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") "
Nov 02 22:29:06.763549 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.763467    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-kube-api-access-gvr2r\" (UniqueName: \"kubernetes.io/projected/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-kube-api-access-gvr2r\") pod \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\" (UID: \"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc\") "
Nov 02 22:29:06.763785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.763755    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-openshift-ca" pod "f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc" (UID: "f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:06.763903 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.763880    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.765492 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.765471    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-xks-ca" pod "f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc" (UID: "f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:06.765575 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.765498    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-kube-api-access-gvr2r" pod "f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc" (UID: "f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc"). InnerVolumeSpecName "kube-api-access-gvr2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:06.865359 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.865323    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvr2r\" (UniqueName: \"kubernetes.io/projected/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-kube-api-access-gvr2r\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.865359 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.865360    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:06.953177 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.953148    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"]
Nov 02 22:29:06.957302 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:06.957279    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c7e71f-openshift-default-549bbf685f-qjjbk"]
Nov 02 22:29:07.444153 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:07.444126    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"
Nov 02 22:29:07.444560 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:07.444168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4" event={"ID":"f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc","Type":"ContainerDied","Data":"3131b9353a64134af0c022b908e2a46b98c252741ace61a8e18d5676020d1536"}
Nov 02 22:29:07.444560 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:07.444223    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3131b9353a64134af0c022b908e2a46b98c252741ace61a8e18d5676020d1536"
Nov 02 22:29:07.844853 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:07.844824    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"]
Nov 02 22:29:08.912926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:08.912889    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" path="/var/lib/kubelet/pods/d474a4c1-c212-4544-8e0c-f9e411b5b219/volumes"
Nov 02 22:29:11.944011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:11.943975    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:29:14.112960 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:14.112929    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:29:17.971924 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:17.971887    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"]
Nov 02 22:29:18.077848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.077813    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"]
Nov 02 22:29:18.457011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.456967    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"]
Nov 02 22:29:18.457576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457557    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy"
Nov 02 22:29:18.457697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457577    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy"
Nov 02 22:29:18.457697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457595    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy"
Nov 02 22:29:18.457697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457602    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy"
Nov 02 22:29:18.457697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457612    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc" containerName="revoke-keys"
Nov 02 22:29:18.457697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457620    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc" containerName="revoke-keys"
Nov 02 22:29:18.457697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457657    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d907fbc4-2060-47ed-81f8-2067ee3f5a1c" containerName="revoke-keys"
Nov 02 22:29:18.457697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457664    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d907fbc4-2060-47ed-81f8-2067ee3f5a1c" containerName="revoke-keys"
Nov 02 22:29:18.458065 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457806    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d907fbc4-2060-47ed-81f8-2067ee3f5a1c" containerName="revoke-keys"
Nov 02 22:29:18.458065 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457824    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy"
Nov 02 22:29:18.458065 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457834    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d474a4c1-c212-4544-8e0c-f9e411b5b219" containerName="istio-proxy"
Nov 02 22:29:18.458065 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.457844    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc" containerName="revoke-keys"
Nov 02 22:29:18.461310 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.461284    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.508387 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.508350    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rthvz\" (UniqueName: \"kubernetes.io/projected/34a59620-b8fe-456f-afb8-eb95c0de11ac-kube-api-access-rthvz\") pod \"maas-api-revoke-sub-994d2ec6-s64j6\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.508565 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.508508    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-994d2ec6-s64j6\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.508642 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.508613    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-994d2ec6-s64j6\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.514515 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.514485    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:29:18.514681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.514656    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:29:18.514781 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.514772    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:29:18.514984 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.514942    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:29:18.519376 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.519167    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"]
Nov 02 22:29:18.609822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.609783    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-994d2ec6-s64j6\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.610013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.609882    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-994d2ec6-s64j6\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.610013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.609948    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rthvz\" (UniqueName: \"kubernetes.io/projected/34a59620-b8fe-456f-afb8-eb95c0de11ac-kube-api-access-rthvz\") pod \"maas-api-revoke-sub-994d2ec6-s64j6\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.610532 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.610507    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-994d2ec6-s64j6\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.612137 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.612090    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-994d2ec6-s64j6\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.666916 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.666880    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rthvz\" (UniqueName: \"kubernetes.io/projected/34a59620-b8fe-456f-afb8-eb95c0de11ac-kube-api-access-rthvz\") pod \"maas-api-revoke-sub-994d2ec6-s64j6\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:18.777588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:18.777558    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:19.075281 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:19.075205    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"]
Nov 02 22:29:19.075690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:19.075431    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy" containerID="cri-o://e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5" gracePeriod=30
Nov 02 22:29:19.186195 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:19.185998    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"]
Nov 02 22:29:19.188493 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:29:19.188466    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a59620_b8fe_456f_afb8_eb95c0de11ac.slice/crio-567eae8e5a910a0c2bc65206c765a564fc7291a7503551898d1abff8cc474d25 WatchSource:0}: Error finding container 567eae8e5a910a0c2bc65206c765a564fc7291a7503551898d1abff8cc474d25: Status 404 returned error can't find the container with id 567eae8e5a910a0c2bc65206c765a564fc7291a7503551898d1abff8cc474d25
Nov 02 22:29:19.521299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:19.521272    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6" event={"ID":"34a59620-b8fe-456f-afb8-eb95c0de11ac","Type":"ContainerStarted","Data":"fb6c6898b5ae63ea969f24898ddb74c1b0d4e0db1c957c1d027e2ef85cc9e662"}
Nov 02 22:29:19.521398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:19.521329    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6" event={"ID":"34a59620-b8fe-456f-afb8-eb95c0de11ac","Type":"ContainerStarted","Data":"567eae8e5a910a0c2bc65206c765a564fc7291a7503551898d1abff8cc474d25"}
Nov 02 22:29:19.635782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:19.635731    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6" podStartSLOduration=1.635719968 podStartE2EDuration="1.635719968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:19.634959388 +0000 UTC m=+4846.898495705" watchObservedRunningTime="2026-11-02 22:29:19.635719968 +0000 UTC m=+4846.899256284"
Nov 02 22:29:20.208167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.208134    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"]
Nov 02 22:29:20.211915 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.211895    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.268389 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.268362    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"]
Nov 02 22:29:20.295992 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.295961    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"]
Nov 02 22:29:20.333240 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.333165    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30fc16c0-jv9jh\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.333408 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.333295    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30fc16c0-jv9jh\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.333408 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.333340    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzk8n\" (UniqueName: \"kubernetes.io/projected/5429792d-7e77-4dff-85e5-60ad855937f8-kube-api-access-vzk8n\") pod \"maas-api-revoke-sub-30fc16c0-jv9jh\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.376676 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.376609    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"]
Nov 02 22:29:20.380674 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.380653    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.393055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.393031    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"]
Nov 02 22:29:20.434735 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.434705    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e235b9dd-pxlrl\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.434867 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.434762    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nq9\" (UniqueName: \"kubernetes.io/projected/7e4750f1-535c-43c2-a353-db31322f9df0-kube-api-access-f5nq9\") pod \"maas-api-revoke-sub-e235b9dd-pxlrl\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.434946 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.434916    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30fc16c0-jv9jh\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.435005 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.434989    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzk8n\" (UniqueName: \"kubernetes.io/projected/5429792d-7e77-4dff-85e5-60ad855937f8-kube-api-access-vzk8n\") pod \"maas-api-revoke-sub-30fc16c0-jv9jh\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.435186 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.435166    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e235b9dd-pxlrl\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.435272 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.435199    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30fc16c0-jv9jh\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.435622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.435596    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30fc16c0-jv9jh\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.438084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.437742    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"]
Nov 02 22:29:20.438084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.438037    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30fc16c0-jv9jh\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.492081 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.492048    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzk8n\" (UniqueName: \"kubernetes.io/projected/5429792d-7e77-4dff-85e5-60ad855937f8-kube-api-access-vzk8n\") pod \"maas-api-revoke-sub-30fc16c0-jv9jh\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.526472 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.526445    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:20.528271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.528241    4528 generic.go:356] "Generic (PLEG): container finished" podID="34a59620-b8fe-456f-afb8-eb95c0de11ac" containerID="fb6c6898b5ae63ea969f24898ddb74c1b0d4e0db1c957c1d027e2ef85cc9e662" exitCode=0
Nov 02 22:29:20.528408 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.528316    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6" event={"ID":"34a59620-b8fe-456f-afb8-eb95c0de11ac","Type":"ContainerDied","Data":"fb6c6898b5ae63ea969f24898ddb74c1b0d4e0db1c957c1d027e2ef85cc9e662"}
Nov 02 22:29:20.536006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.535982    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e235b9dd-pxlrl\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.536139 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.536032    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e235b9dd-pxlrl\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.536139 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.536071    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nq9\" (UniqueName: \"kubernetes.io/projected/7e4750f1-535c-43c2-a353-db31322f9df0-kube-api-access-f5nq9\") pod \"maas-api-revoke-sub-e235b9dd-pxlrl\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.536630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.536609    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e235b9dd-pxlrl\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.538362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.538341    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e235b9dd-pxlrl\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.599220 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.598935    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nq9\" (UniqueName: \"kubernetes.io/projected/7e4750f1-535c-43c2-a353-db31322f9df0-kube-api-access-f5nq9\") pod \"maas-api-revoke-sub-e235b9dd-pxlrl\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.699159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.693813    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:20.896861 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:29:20.896833    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5429792d_7e77_4dff_85e5_60ad855937f8.slice/crio-e022c73477bc8c3fe879c081bfd23accd771597e95b50ca61c93b31d57bf5606 WatchSource:0}: Error finding container e022c73477bc8c3fe879c081bfd23accd771597e95b50ca61c93b31d57bf5606: Status 404 returned error can't find the container with id e022c73477bc8c3fe879c081bfd23accd771597e95b50ca61c93b31d57bf5606
Nov 02 22:29:20.903864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:20.903839    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"]
Nov 02 22:29:21.076968 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:29:21.076942    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4750f1_535c_43c2_a353_db31322f9df0.slice/crio-0f70b1c14a6fead24e1e7e5163a6a9136f6a25b9417b2110105141498aab419b WatchSource:0}: Error finding container 0f70b1c14a6fead24e1e7e5163a6a9136f6a25b9417b2110105141498aab419b: Status 404 returned error can't find the container with id 0f70b1c14a6fead24e1e7e5163a6a9136f6a25b9417b2110105141498aab419b
Nov 02 22:29:21.079720 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.079694    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"]
Nov 02 22:29:21.523068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.523034    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main" containerID="cri-o://ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0" gracePeriod=45
Nov 02 22:29:21.536248 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.536223    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl" event={"ID":"7e4750f1-535c-43c2-a353-db31322f9df0","Type":"ContainerStarted","Data":"643b1b48919b767c6515a64c6f9c18f659b9cb12a59cf982e38565c4edbac1e6"}
Nov 02 22:29:21.536372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.536269    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl" event={"ID":"7e4750f1-535c-43c2-a353-db31322f9df0","Type":"ContainerStarted","Data":"0f70b1c14a6fead24e1e7e5163a6a9136f6a25b9417b2110105141498aab419b"}
Nov 02 22:29:21.538186 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.538162    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh" event={"ID":"5429792d-7e77-4dff-85e5-60ad855937f8","Type":"ContainerStarted","Data":"31cd6554c578151697d288d472b59e3b512350a1927eb606dc81c169bc92b721"}
Nov 02 22:29:21.538300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.538210    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh" event={"ID":"5429792d-7e77-4dff-85e5-60ad855937f8","Type":"ContainerStarted","Data":"e022c73477bc8c3fe879c081bfd23accd771597e95b50ca61c93b31d57bf5606"}
Nov 02 22:29:21.655204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.655144    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl" podStartSLOduration=1.6551258949999998 podStartE2EDuration="1.655125895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:21.651550438 +0000 UTC m=+4848.915086750" watchObservedRunningTime="2026-11-02 22:29:21.655125895 +0000 UTC m=+4848.918662213"
Nov 02 22:29:21.678174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.678150    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:21.750163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.750132    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-openshift-ca\") pod \"34a59620-b8fe-456f-afb8-eb95c0de11ac\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") "
Nov 02 22:29:21.750341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.750232    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a59620-b8fe-456f-afb8-eb95c0de11ac-kube-api-access-rthvz\" (UniqueName: \"kubernetes.io/projected/34a59620-b8fe-456f-afb8-eb95c0de11ac-kube-api-access-rthvz\") pod \"34a59620-b8fe-456f-afb8-eb95c0de11ac\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") "
Nov 02 22:29:21.750341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.750317    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-xks-ca\") pod \"34a59620-b8fe-456f-afb8-eb95c0de11ac\" (UID: \"34a59620-b8fe-456f-afb8-eb95c0de11ac\") "
Nov 02 22:29:21.750533 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.750507    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-openshift-ca" pod "34a59620-b8fe-456f-afb8-eb95c0de11ac" (UID: "34a59620-b8fe-456f-afb8-eb95c0de11ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:21.750790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.750768    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:21.752315 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.752288    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-xks-ca" pod "34a59620-b8fe-456f-afb8-eb95c0de11ac" (UID: "34a59620-b8fe-456f-afb8-eb95c0de11ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:21.752476 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.752457    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a59620-b8fe-456f-afb8-eb95c0de11ac-kube-api-access-rthvz" pod "34a59620-b8fe-456f-afb8-eb95c0de11ac" (UID: "34a59620-b8fe-456f-afb8-eb95c0de11ac"). InnerVolumeSpecName "kube-api-access-rthvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:21.766480 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.766440    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh" podStartSLOduration=1.766428129 podStartE2EDuration="1.766428129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:21.76586289 +0000 UTC m=+4849.029399233" watchObservedRunningTime="2026-11-02 22:29:21.766428129 +0000 UTC m=+4849.029964448"
Nov 02 22:29:21.796480 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.796454    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 22:29:21.851532 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.851495    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-model-cache\") pod \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") "
Nov 02 22:29:21.851705 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.851659    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-home\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-home\") pod \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") "
Nov 02 22:29:21.851785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.851705    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-dshm\") pod \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") "
Nov 02 22:29:21.851847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.851792    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-model-cache" pod "182772ef-d7fb-465f-94f6-20e4fc63e7f5" (UID: "182772ef-d7fb-465f-94f6-20e4fc63e7f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:21.851847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.851810    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tmp-dir\") pod \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") "
Nov 02 22:29:21.851963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.851884    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tls-certs\") pod \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") "
Nov 02 22:29:21.851963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.851901    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-home" pod "182772ef-d7fb-465f-94f6-20e4fc63e7f5" (UID: "182772ef-d7fb-465f-94f6-20e4fc63e7f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:21.851963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.851917    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182772ef-d7fb-465f-94f6-20e4fc63e7f5-kube-api-access-sv7cb\" (UniqueName: \"kubernetes.io/projected/182772ef-d7fb-465f-94f6-20e4fc63e7f5-kube-api-access-sv7cb\") pod \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\" (UID: \"182772ef-d7fb-465f-94f6-20e4fc63e7f5\") "
Nov 02 22:29:21.852141 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.852090    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tmp-dir" pod "182772ef-d7fb-465f-94f6-20e4fc63e7f5" (UID: "182772ef-d7fb-465f-94f6-20e4fc63e7f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:21.852323 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.852305    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:21.852396 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.852327    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:21.852396 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.852343    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rthvz\" (UniqueName: \"kubernetes.io/projected/34a59620-b8fe-456f-afb8-eb95c0de11ac-kube-api-access-rthvz\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:21.852396 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.852356    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:21.852396 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.852370    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a59620-b8fe-456f-afb8-eb95c0de11ac-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:21.854250 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.854219    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-dshm" pod "182772ef-d7fb-465f-94f6-20e4fc63e7f5" (UID: "182772ef-d7fb-465f-94f6-20e4fc63e7f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:21.854368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.854351    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tls-certs" pod "182772ef-d7fb-465f-94f6-20e4fc63e7f5" (UID: "182772ef-d7fb-465f-94f6-20e4fc63e7f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:21.854490 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.854469    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182772ef-d7fb-465f-94f6-20e4fc63e7f5-kube-api-access-sv7cb" pod "182772ef-d7fb-465f-94f6-20e4fc63e7f5" (UID: "182772ef-d7fb-465f-94f6-20e4fc63e7f5"). InnerVolumeSpecName "kube-api-access-sv7cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:21.953857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.953823    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182772ef-d7fb-465f-94f6-20e4fc63e7f5-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:21.953857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.953852    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv7cb\" (UniqueName: \"kubernetes.io/projected/182772ef-d7fb-465f-94f6-20e4fc63e7f5-kube-api-access-sv7cb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:21.953857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:21.953862    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182772ef-d7fb-465f-94f6-20e4fc63e7f5-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:22.566852 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.566817    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6" event={"ID":"34a59620-b8fe-456f-afb8-eb95c0de11ac","Type":"ContainerDied","Data":"567eae8e5a910a0c2bc65206c765a564fc7291a7503551898d1abff8cc474d25"}
Nov 02 22:29:22.571248 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.571201    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="567eae8e5a910a0c2bc65206c765a564fc7291a7503551898d1abff8cc474d25"
Nov 02 22:29:22.572003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.571975    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"
Nov 02 22:29:22.574296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.574266    4528 generic.go:356] "Generic (PLEG): container finished" podID="7e4750f1-535c-43c2-a353-db31322f9df0" containerID="643b1b48919b767c6515a64c6f9c18f659b9cb12a59cf982e38565c4edbac1e6" exitCode=0
Nov 02 22:29:22.574446 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.574354    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl" event={"ID":"7e4750f1-535c-43c2-a353-db31322f9df0","Type":"ContainerDied","Data":"643b1b48919b767c6515a64c6f9c18f659b9cb12a59cf982e38565c4edbac1e6"}
Nov 02 22:29:22.576155 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.576124    4528 generic.go:356] "Generic (PLEG): container finished" podID="5429792d-7e77-4dff-85e5-60ad855937f8" containerID="31cd6554c578151697d288d472b59e3b512350a1927eb606dc81c169bc92b721" exitCode=0
Nov 02 22:29:22.576286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.576153    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh" event={"ID":"5429792d-7e77-4dff-85e5-60ad855937f8","Type":"ContainerDied","Data":"31cd6554c578151697d288d472b59e3b512350a1927eb606dc81c169bc92b721"}
Nov 02 22:29:22.577908 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.577886    4528 generic.go:356] "Generic (PLEG): container finished" podID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerID="ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0" exitCode=0
Nov 02 22:29:22.577998 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.577908    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" event={"ID":"182772ef-d7fb-465f-94f6-20e4fc63e7f5","Type":"ContainerDied","Data":"ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0"}
Nov 02 22:29:22.577998 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.577924    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"
Nov 02 22:29:22.577998 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.577943    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6" event={"ID":"182772ef-d7fb-465f-94f6-20e4fc63e7f5","Type":"ContainerDied","Data":"f9c4eb6956128cb6e897a067ff20e0c1d92d0c9e746316ebfb0a6548ce0d95bf"}
Nov 02 22:29:22.577998 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.577964    4528 scope.go:122] "RemoveContainer" containerID="ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0"
Nov 02 22:29:22.587648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.587633    4528 scope.go:122] "RemoveContainer" containerID="ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0"
Nov 02 22:29:22.587877 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:22.587858    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0\": container with ID starting with ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0 not found: ID does not exist" containerID="ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0"
Nov 02 22:29:22.587923 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.587885    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0"} err="failed to get container status \"ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0\": rpc error: code = NotFound desc = could not find container \"ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0\": container with ID starting with ea7dd9f3f548078975b81051e04cd47fadc270656080abd9040f612a586082e0 not found: ID does not exist"
Nov 02 22:29:22.799059 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.799029    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"]
Nov 02 22:29:22.803948 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.803923    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/premium-w2-6fa533-kserve-5c54778dd-6q2h6"]
Nov 02 22:29:22.865312 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.865229    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main" containerID="cri-o://1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e" gracePeriod=45
Nov 02 22:29:22.913542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.913513    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" path="/var/lib/kubelet/pods/182772ef-d7fb-465f-94f6-20e4fc63e7f5/volumes"
Nov 02 22:29:22.972471 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:22.972441    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4" podUID="f7bc743a-7ada-48a6-95be-d6adfedbac23" containerName="payload-processing" containerID="cri-o://840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713" gracePeriod=25
Nov 02 22:29:23.078976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.078941    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7" podUID="e9e8e764-309c-4788-8268-b27f2dc8f290" containerName="payload-pre-processing" containerID="cri-o://e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c" gracePeriod=25
Nov 02 22:29:23.125340 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.125276    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 22:29:23.170811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.170773    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-model-cache\") pod \"f21efa86-2527-440b-acd0-657466f62f1e\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") "
Nov 02 22:29:23.170964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.170854    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f21efa86-2527-440b-acd0-657466f62f1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f21efa86-2527-440b-acd0-657466f62f1e-tls-certs\") pod \"f21efa86-2527-440b-acd0-657466f62f1e\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") "
Nov 02 22:29:23.170964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.170922    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-tmp-dir\") pod \"f21efa86-2527-440b-acd0-657466f62f1e\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") "
Nov 02 22:29:23.170964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.170955    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-dshm\") pod \"f21efa86-2527-440b-acd0-657466f62f1e\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") "
Nov 02 22:29:23.171162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.170997    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21efa86-2527-440b-acd0-657466f62f1e-kube-api-access-m2gmk\" (UniqueName: \"kubernetes.io/projected/f21efa86-2527-440b-acd0-657466f62f1e-kube-api-access-m2gmk\") pod \"f21efa86-2527-440b-acd0-657466f62f1e\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") "
Nov 02 22:29:23.171162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.171086    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-home\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-home\") pod \"f21efa86-2527-440b-acd0-657466f62f1e\" (UID: \"f21efa86-2527-440b-acd0-657466f62f1e\") "
Nov 02 22:29:23.171290 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.171153    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-model-cache" pod "f21efa86-2527-440b-acd0-657466f62f1e" (UID: "f21efa86-2527-440b-acd0-657466f62f1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:23.171556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.171527    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-home" pod "f21efa86-2527-440b-acd0-657466f62f1e" (UID: "f21efa86-2527-440b-acd0-657466f62f1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:23.171712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.171591    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-tmp-dir" pod "f21efa86-2527-440b-acd0-657466f62f1e" (UID: "f21efa86-2527-440b-acd0-657466f62f1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:23.171712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.171591    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.174355 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.174325    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-dshm" pod "f21efa86-2527-440b-acd0-657466f62f1e" (UID: "f21efa86-2527-440b-acd0-657466f62f1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:23.174462 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.174386    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21efa86-2527-440b-acd0-657466f62f1e-tls-certs" pod "f21efa86-2527-440b-acd0-657466f62f1e" (UID: "f21efa86-2527-440b-acd0-657466f62f1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:23.174462 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.174426    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21efa86-2527-440b-acd0-657466f62f1e-kube-api-access-m2gmk" pod "f21efa86-2527-440b-acd0-657466f62f1e" (UID: "f21efa86-2527-440b-acd0-657466f62f1e"). InnerVolumeSpecName "kube-api-access-m2gmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:23.221566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.221541    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 22:29:23.272778 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.272744    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bc743a-7ada-48a6-95be-d6adfedbac23-kube-api-access-rh8h7\" (UniqueName: \"kubernetes.io/projected/f7bc743a-7ada-48a6-95be-d6adfedbac23-kube-api-access-rh8h7\") pod \"f7bc743a-7ada-48a6-95be-d6adfedbac23\" (UID: \"f7bc743a-7ada-48a6-95be-d6adfedbac23\") "
Nov 02 22:29:23.272952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.272820    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bc743a-7ada-48a6-95be-d6adfedbac23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc743a-7ada-48a6-95be-d6adfedbac23-plugins-config-volume\") pod \"f7bc743a-7ada-48a6-95be-d6adfedbac23\" (UID: \"f7bc743a-7ada-48a6-95be-d6adfedbac23\") "
Nov 02 22:29:23.273223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.273201    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21efa86-2527-440b-acd0-657466f62f1e-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.273377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.273249    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.273377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.273264    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.273377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.273278    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gmk\" (UniqueName: \"kubernetes.io/projected/f21efa86-2527-440b-acd0-657466f62f1e-kube-api-access-m2gmk\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.273377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.273292    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21efa86-2527-440b-acd0-657466f62f1e-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.273572 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.273461    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bc743a-7ada-48a6-95be-d6adfedbac23-plugins-config-volume" pod "f7bc743a-7ada-48a6-95be-d6adfedbac23" (UID: "f7bc743a-7ada-48a6-95be-d6adfedbac23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:23.275562 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.275542    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bc743a-7ada-48a6-95be-d6adfedbac23-kube-api-access-rh8h7" pod "f7bc743a-7ada-48a6-95be-d6adfedbac23" (UID: "f7bc743a-7ada-48a6-95be-d6adfedbac23"). InnerVolumeSpecName "kube-api-access-rh8h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:23.374259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.374227    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh8h7\" (UniqueName: \"kubernetes.io/projected/f7bc743a-7ada-48a6-95be-d6adfedbac23-kube-api-access-rh8h7\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.374259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.374255    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc743a-7ada-48a6-95be-d6adfedbac23-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.586163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.586101    4528 generic.go:356] "Generic (PLEG): container finished" podID="f7bc743a-7ada-48a6-95be-d6adfedbac23" containerID="840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713" exitCode=0
Nov 02 22:29:23.586163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.586156    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"
Nov 02 22:29:23.586640 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.586208    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4" event={"ID":"f7bc743a-7ada-48a6-95be-d6adfedbac23","Type":"ContainerDied","Data":"840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713"}
Nov 02 22:29:23.586640 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.586254    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4" event={"ID":"f7bc743a-7ada-48a6-95be-d6adfedbac23","Type":"ContainerDied","Data":"c7b784839250f1517a3c550ebac3383e0914b18a4b2015c8afe6a47c369db493"}
Nov 02 22:29:23.586640 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.586277    4528 scope.go:122] "RemoveContainer" containerID="840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713"
Nov 02 22:29:23.587654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.587635    4528 generic.go:356] "Generic (PLEG): container finished" podID="f21efa86-2527-440b-acd0-657466f62f1e" containerID="1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e" exitCode=0
Nov 02 22:29:23.587744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.587671    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" event={"ID":"f21efa86-2527-440b-acd0-657466f62f1e","Type":"ContainerDied","Data":"1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e"}
Nov 02 22:29:23.587744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.587690    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"
Nov 02 22:29:23.587744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.587703    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf" event={"ID":"f21efa86-2527-440b-acd0-657466f62f1e","Type":"ContainerDied","Data":"73a2a5db99f5626b782a0e2ca1ced4a6dfe752dbfe135e479baa0802395899d2"}
Nov 02 22:29:23.597802 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.597779    4528 scope.go:122] "RemoveContainer" containerID="840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713"
Nov 02 22:29:23.598103 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:23.598078    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713\": container with ID starting with 840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713 not found: ID does not exist" containerID="840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713"
Nov 02 22:29:23.598251 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.598171    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713"} err="failed to get container status \"840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713\": rpc error: code = NotFound desc = could not find container \"840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713\": container with ID starting with 840e9c7f792cd6179f9b735eee0cd2bcb3a399400de0a3cc66424d49de864713 not found: ID does not exist"
Nov 02 22:29:23.598251 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.598197    4528 scope.go:122] "RemoveContainer" containerID="1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e"
Nov 02 22:29:23.609706 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.609679    4528 scope.go:122] "RemoveContainer" containerID="1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e"
Nov 02 22:29:23.609978 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:23.609954    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e\": container with ID starting with 1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e not found: ID does not exist" containerID="1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e"
Nov 02 22:29:23.610070 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.609988    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e"} err="failed to get container status \"1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e\": rpc error: code = NotFound desc = could not find container \"1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e\": container with ID starting with 1b0639b688951c62d0dac6077e955ac2173cf7eecccc7d4bf2947b1fafe11b2e not found: ID does not exist"
Nov 02 22:29:23.751949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.751923    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"]
Nov 02 22:29:23.754736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.754714    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-2-w1-5f765a-kserve-56cc44bb47-7tvlf"]
Nov 02 22:29:23.771336 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.771314    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:23.773371 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.773354    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:23.862436 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.862364    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"]
Nov 02 22:29:23.868240 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.868216    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6fa533-58b87d55-zwcz4"]
Nov 02 22:29:23.879258 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.879240    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-openshift-ca\") pod \"5429792d-7e77-4dff-85e5-60ad855937f8\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") "
Nov 02 22:29:23.879346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.879275    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-xks-ca\") pod \"7e4750f1-535c-43c2-a353-db31322f9df0\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") "
Nov 02 22:29:23.879346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.879299    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5429792d-7e77-4dff-85e5-60ad855937f8-kube-api-access-vzk8n\" (UniqueName: \"kubernetes.io/projected/5429792d-7e77-4dff-85e5-60ad855937f8-kube-api-access-vzk8n\") pod \"5429792d-7e77-4dff-85e5-60ad855937f8\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") "
Nov 02 22:29:23.879475 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.879463    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4750f1-535c-43c2-a353-db31322f9df0-kube-api-access-f5nq9\" (UniqueName: \"kubernetes.io/projected/7e4750f1-535c-43c2-a353-db31322f9df0-kube-api-access-f5nq9\") pod \"7e4750f1-535c-43c2-a353-db31322f9df0\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") "
Nov 02 22:29:23.879521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.879493    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-openshift-ca\") pod \"7e4750f1-535c-43c2-a353-db31322f9df0\" (UID: \"7e4750f1-535c-43c2-a353-db31322f9df0\") "
Nov 02 22:29:23.879565 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.879525    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-xks-ca\") pod \"5429792d-7e77-4dff-85e5-60ad855937f8\" (UID: \"5429792d-7e77-4dff-85e5-60ad855937f8\") "
Nov 02 22:29:23.879665 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.879636    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-openshift-ca" pod "5429792d-7e77-4dff-85e5-60ad855937f8" (UID: "5429792d-7e77-4dff-85e5-60ad855937f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:23.879906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.879876    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-openshift-ca" pod "7e4750f1-535c-43c2-a353-db31322f9df0" (UID: "7e4750f1-535c-43c2-a353-db31322f9df0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:23.880063 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.880042    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.880156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.880069    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.881512 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.881486    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-xks-ca" pod "7e4750f1-535c-43c2-a353-db31322f9df0" (UID: "7e4750f1-535c-43c2-a353-db31322f9df0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:23.881602 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.881519    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-xks-ca" pod "5429792d-7e77-4dff-85e5-60ad855937f8" (UID: "5429792d-7e77-4dff-85e5-60ad855937f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:23.881602 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.881532    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4750f1-535c-43c2-a353-db31322f9df0-kube-api-access-f5nq9" pod "7e4750f1-535c-43c2-a353-db31322f9df0" (UID: "7e4750f1-535c-43c2-a353-db31322f9df0"). InnerVolumeSpecName "kube-api-access-f5nq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:23.881602 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.881554    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5429792d-7e77-4dff-85e5-60ad855937f8-kube-api-access-vzk8n" pod "5429792d-7e77-4dff-85e5-60ad855937f8" (UID: "5429792d-7e77-4dff-85e5-60ad855937f8"). InnerVolumeSpecName "kube-api-access-vzk8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:23.981989 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.981688    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429792d-7e77-4dff-85e5-60ad855937f8-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.981989 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.981718    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4750f1-535c-43c2-a353-db31322f9df0-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.981989 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.981732    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzk8n\" (UniqueName: \"kubernetes.io/projected/5429792d-7e77-4dff-85e5-60ad855937f8-kube-api-access-vzk8n\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:23.981989 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:23.981746    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5nq9\" (UniqueName: \"kubernetes.io/projected/7e4750f1-535c-43c2-a353-db31322f9df0-kube-api-access-f5nq9\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.243814 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.243778    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7" podUID="e9e8e764-309c-4788-8268-b27f2dc8f290" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:29:24.317399 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.317374    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 22:29:24.385544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385514    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-socket\") pod \"b33afd9e-0e68-4231-910d-28348af5ab93\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") "
Nov 02 22:29:24.385697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385568    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33afd9e-0e68-4231-910d-28348af5ab93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33afd9e-0e68-4231-910d-28348af5ab93-istiod-ca-cert\") pod \"b33afd9e-0e68-4231-910d-28348af5ab93\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") "
Nov 02 22:29:24.385697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385594    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b33afd9e-0e68-4231-910d-28348af5ab93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33afd9e-0e68-4231-910d-28348af5ab93-istio-podinfo\") pod \"b33afd9e-0e68-4231-910d-28348af5ab93\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") "
Nov 02 22:29:24.385697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385626    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-data\") pod \"b33afd9e-0e68-4231-910d-28348af5ab93\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") "
Nov 02 22:29:24.385697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385657    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-istio-token\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-istio-token\") pod \"b33afd9e-0e68-4231-910d-28348af5ab93\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") "
Nov 02 22:29:24.385886 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385792    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-certs\") pod \"b33afd9e-0e68-4231-910d-28348af5ab93\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") "
Nov 02 22:29:24.385886 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385816    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-socket" pod "b33afd9e-0e68-4231-910d-28348af5ab93" (UID: "b33afd9e-0e68-4231-910d-28348af5ab93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:24.386003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385894    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-kube-api-access-mgbh4\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-kube-api-access-mgbh4\") pod \"b33afd9e-0e68-4231-910d-28348af5ab93\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") "
Nov 02 22:29:24.386003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385944    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-data" pod "b33afd9e-0e68-4231-910d-28348af5ab93" (UID: "b33afd9e-0e68-4231-910d-28348af5ab93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:24.386003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.385945    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-credential-socket\") pod \"b33afd9e-0e68-4231-910d-28348af5ab93\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") "
Nov 02 22:29:24.386263 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.386067    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-envoy\") pod \"b33afd9e-0e68-4231-910d-28348af5ab93\" (UID: \"b33afd9e-0e68-4231-910d-28348af5ab93\") "
Nov 02 22:29:24.386263 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.386076    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-certs" pod "b33afd9e-0e68-4231-910d-28348af5ab93" (UID: "b33afd9e-0e68-4231-910d-28348af5ab93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:24.386263 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.386145    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33afd9e-0e68-4231-910d-28348af5ab93-istiod-ca-cert" pod "b33afd9e-0e68-4231-910d-28348af5ab93" (UID: "b33afd9e-0e68-4231-910d-28348af5ab93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:24.386263 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.386150    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-credential-socket" pod "b33afd9e-0e68-4231-910d-28348af5ab93" (UID: "b33afd9e-0e68-4231-910d-28348af5ab93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:24.386526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.386505    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.386526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.386527    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.386723 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.386541    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.386723 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.386555    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.386723 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.386568    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33afd9e-0e68-4231-910d-28348af5ab93-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.387973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.387953    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-istio-token" pod "b33afd9e-0e68-4231-910d-28348af5ab93" (UID: "b33afd9e-0e68-4231-910d-28348af5ab93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:24.388057 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.388004    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-envoy" pod "b33afd9e-0e68-4231-910d-28348af5ab93" (UID: "b33afd9e-0e68-4231-910d-28348af5ab93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:24.388259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.388234    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b33afd9e-0e68-4231-910d-28348af5ab93-istio-podinfo" pod "b33afd9e-0e68-4231-910d-28348af5ab93" (UID: "b33afd9e-0e68-4231-910d-28348af5ab93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:29:24.388259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.388253    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-kube-api-access-mgbh4" pod "b33afd9e-0e68-4231-910d-28348af5ab93" (UID: "b33afd9e-0e68-4231-910d-28348af5ab93"). InnerVolumeSpecName "kube-api-access-mgbh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:24.487910 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.487841    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33afd9e-0e68-4231-910d-28348af5ab93-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.487910 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.487866    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33afd9e-0e68-4231-910d-28348af5ab93-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.487910 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.487875    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.487910 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.487883    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgbh4\" (UniqueName: \"kubernetes.io/projected/b33afd9e-0e68-4231-910d-28348af5ab93-kube-api-access-mgbh4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:24.594719 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.594687    4528 generic.go:356] "Generic (PLEG): container finished" podID="b33afd9e-0e68-4231-910d-28348af5ab93" containerID="e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5" exitCode=0
Nov 02 22:29:24.595155 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.594735    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"
Nov 02 22:29:24.595155 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.594772    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" event={"ID":"b33afd9e-0e68-4231-910d-28348af5ab93","Type":"ContainerDied","Data":"e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5"}
Nov 02 22:29:24.595155 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.594833    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq" event={"ID":"b33afd9e-0e68-4231-910d-28348af5ab93","Type":"ContainerDied","Data":"9437d2ed0c81e928b78a999d2c96f8a4495eb3acfeab95362e90f00fd93b749d"}
Nov 02 22:29:24.595155 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.594868    4528 scope.go:122] "RemoveContainer" containerID="e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5"
Nov 02 22:29:24.598536 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.598513    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"
Nov 02 22:29:24.598668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.598530    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl" event={"ID":"7e4750f1-535c-43c2-a353-db31322f9df0","Type":"ContainerDied","Data":"0f70b1c14a6fead24e1e7e5163a6a9136f6a25b9417b2110105141498aab419b"}
Nov 02 22:29:24.598668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.598573    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f70b1c14a6fead24e1e7e5163a6a9136f6a25b9417b2110105141498aab419b"
Nov 02 22:29:24.599987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.599961    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"
Nov 02 22:29:24.599987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.599972    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh" event={"ID":"5429792d-7e77-4dff-85e5-60ad855937f8","Type":"ContainerDied","Data":"e022c73477bc8c3fe879c081bfd23accd771597e95b50ca61c93b31d57bf5606"}
Nov 02 22:29:24.600177 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.600014    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e022c73477bc8c3fe879c081bfd23accd771597e95b50ca61c93b31d57bf5606"
Nov 02 22:29:24.605645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.605628    4528 scope.go:122] "RemoveContainer" containerID="c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b"
Nov 02 22:29:24.615379 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.615360    4528 scope.go:122] "RemoveContainer" containerID="e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5"
Nov 02 22:29:24.615673 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:24.615653    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5\": container with ID starting with e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5 not found: ID does not exist" containerID="e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5"
Nov 02 22:29:24.615734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.615682    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5"} err="failed to get container status \"e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5\": rpc error: code = NotFound desc = could not find container \"e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5\": container with ID starting with e0e39215a82c05ff0de787a8ee5fb5a874cf1552c7a5e95b733539661fa9b0d5 not found: ID does not exist"
Nov 02 22:29:24.615734 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.615697    4528 scope.go:122] "RemoveContainer" containerID="c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b"
Nov 02 22:29:24.615932 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:24.615911    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b\": container with ID starting with c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b not found: ID does not exist" containerID="c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b"
Nov 02 22:29:24.615979 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.615938    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b"} err="failed to get container status \"c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b\": rpc error: code = NotFound desc = could not find container \"c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b\": container with ID starting with c8df65acbdcb72fa5c558c4c49f1067c6f71ddb9fc99599bdd4fa6165996f25b not found: ID does not exist"
Nov 02 22:29:24.814441 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.814405    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"]
Nov 02 22:29:24.819969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.819942    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7cde84-openshift-default-85987bf5b5-mh4nq"]
Nov 02 22:29:24.914390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.914355    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" path="/var/lib/kubelet/pods/b33afd9e-0e68-4231-910d-28348af5ab93/volumes"
Nov 02 22:29:24.914819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.914804    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21efa86-2527-440b-acd0-657466f62f1e" path="/var/lib/kubelet/pods/f21efa86-2527-440b-acd0-657466f62f1e/volumes"
Nov 02 22:29:24.915173 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:24.915161    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bc743a-7ada-48a6-95be-d6adfedbac23" path="/var/lib/kubelet/pods/f7bc743a-7ada-48a6-95be-d6adfedbac23/volumes"
Nov 02 22:29:25.296634 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.296598    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx" podUID="b092804d-6683-4b47-9d26-ad6d9d2df062" containerName="payload-processing" containerID="cri-o://b4b203b8a7f04f7185994362b19c739e1fafc6d1749aff9b6a732ecf21ede65a" gracePeriod=25
Nov 02 22:29:25.393478 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.393442    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl" podUID="2f052551-4d9f-455d-b26f-8f63f00a1885" containerName="payload-pre-processing" containerID="cri-o://ceb5b58190e000694c62ea2ce887286c1c998602e8c9afba2537f276ac4a82c7" gracePeriod=25
Nov 02 22:29:25.611095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.611059    4528 generic.go:356] "Generic (PLEG): container finished" podID="2f052551-4d9f-455d-b26f-8f63f00a1885" containerID="ceb5b58190e000694c62ea2ce887286c1c998602e8c9afba2537f276ac4a82c7" exitCode=0
Nov 02 22:29:25.612840 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.611339    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl" event={"ID":"2f052551-4d9f-455d-b26f-8f63f00a1885","Type":"ContainerDied","Data":"ceb5b58190e000694c62ea2ce887286c1c998602e8c9afba2537f276ac4a82c7"}
Nov 02 22:29:25.617649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.617624    4528 generic.go:356] "Generic (PLEG): container finished" podID="b092804d-6683-4b47-9d26-ad6d9d2df062" containerID="b4b203b8a7f04f7185994362b19c739e1fafc6d1749aff9b6a732ecf21ede65a" exitCode=0
Nov 02 22:29:25.617877 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.617858    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx" event={"ID":"b092804d-6683-4b47-9d26-ad6d9d2df062","Type":"ContainerDied","Data":"b4b203b8a7f04f7185994362b19c739e1fafc6d1749aff9b6a732ecf21ede65a"}
Nov 02 22:29:25.677068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.677046    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 22:29:25.709886 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.709865    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 22:29:25.800084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.800054    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f052551-4d9f-455d-b26f-8f63f00a1885-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f052551-4d9f-455d-b26f-8f63f00a1885-plugins-config-volume\") pod \"2f052551-4d9f-455d-b26f-8f63f00a1885\" (UID: \"2f052551-4d9f-455d-b26f-8f63f00a1885\") "
Nov 02 22:29:25.800252 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.800139    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b092804d-6683-4b47-9d26-ad6d9d2df062-kube-api-access-mgzb4\" (UniqueName: \"kubernetes.io/projected/b092804d-6683-4b47-9d26-ad6d9d2df062-kube-api-access-mgzb4\") pod \"b092804d-6683-4b47-9d26-ad6d9d2df062\" (UID: \"b092804d-6683-4b47-9d26-ad6d9d2df062\") "
Nov 02 22:29:25.800252 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.800162    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f052551-4d9f-455d-b26f-8f63f00a1885-kube-api-access-6lmld\" (UniqueName: \"kubernetes.io/projected/2f052551-4d9f-455d-b26f-8f63f00a1885-kube-api-access-6lmld\") pod \"2f052551-4d9f-455d-b26f-8f63f00a1885\" (UID: \"2f052551-4d9f-455d-b26f-8f63f00a1885\") "
Nov 02 22:29:25.800252 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.800205    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b092804d-6683-4b47-9d26-ad6d9d2df062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092804d-6683-4b47-9d26-ad6d9d2df062-plugins-config-volume\") pod \"b092804d-6683-4b47-9d26-ad6d9d2df062\" (UID: \"b092804d-6683-4b47-9d26-ad6d9d2df062\") "
Nov 02 22:29:25.800638 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.800606    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f052551-4d9f-455d-b26f-8f63f00a1885-plugins-config-volume" pod "2f052551-4d9f-455d-b26f-8f63f00a1885" (UID: "2f052551-4d9f-455d-b26f-8f63f00a1885"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:25.800776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.800625    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b092804d-6683-4b47-9d26-ad6d9d2df062-plugins-config-volume" pod "b092804d-6683-4b47-9d26-ad6d9d2df062" (UID: "b092804d-6683-4b47-9d26-ad6d9d2df062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:25.802466 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.802440    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b092804d-6683-4b47-9d26-ad6d9d2df062-kube-api-access-mgzb4" pod "b092804d-6683-4b47-9d26-ad6d9d2df062" (UID: "b092804d-6683-4b47-9d26-ad6d9d2df062"). InnerVolumeSpecName "kube-api-access-mgzb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:25.802667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.802644    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f052551-4d9f-455d-b26f-8f63f00a1885-kube-api-access-6lmld" pod "2f052551-4d9f-455d-b26f-8f63f00a1885" (UID: "2f052551-4d9f-455d-b26f-8f63f00a1885"). InnerVolumeSpecName "kube-api-access-6lmld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:25.901506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.901474    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f052551-4d9f-455d-b26f-8f63f00a1885-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:25.901506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.901502    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgzb4\" (UniqueName: \"kubernetes.io/projected/b092804d-6683-4b47-9d26-ad6d9d2df062-kube-api-access-mgzb4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:25.901506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.901511    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lmld\" (UniqueName: \"kubernetes.io/projected/2f052551-4d9f-455d-b26f-8f63f00a1885-kube-api-access-6lmld\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:25.901736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:25.901519    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092804d-6683-4b47-9d26-ad6d9d2df062-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:26.283930 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.283898    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"]
Nov 02 22:29:26.381286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.381258    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"]
Nov 02 22:29:26.624953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.624870    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"
Nov 02 22:29:26.624953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.624897    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx" event={"ID":"b092804d-6683-4b47-9d26-ad6d9d2df062","Type":"ContainerDied","Data":"a055bc09fe22e156beb47ea55df23d29c6205f03a8519736171f86a25963259e"}
Nov 02 22:29:26.625528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.624970    4528 scope.go:122] "RemoveContainer" containerID="b4b203b8a7f04f7185994362b19c739e1fafc6d1749aff9b6a732ecf21ede65a"
Nov 02 22:29:26.626428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.626404    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl" event={"ID":"2f052551-4d9f-455d-b26f-8f63f00a1885","Type":"ContainerDied","Data":"908be547f0e44ecc7ba6ece73f10f35dab9c9bd609a638fa9a9b1a2a020b0808"}
Nov 02 22:29:26.626428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.626425    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"
Nov 02 22:29:26.637695 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.637677    4528 scope.go:122] "RemoveContainer" containerID="ceb5b58190e000694c62ea2ce887286c1c998602e8c9afba2537f276ac4a82c7"
Nov 02 22:29:26.798420 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.798386    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"]
Nov 02 22:29:26.802986 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.802955    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f765a-5d77cd8677-htgkx"]
Nov 02 22:29:26.914249 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.914179    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b092804d-6683-4b47-9d26-ad6d9d2df062" path="/var/lib/kubelet/pods/b092804d-6683-4b47-9d26-ad6d9d2df062/volumes"
Nov 02 22:29:26.914489 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.914476    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"]
Nov 02 22:29:26.914534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:26.914505    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f765a-77f744b95-n45vl"]
Nov 02 22:29:28.912470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:28.912439    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f052551-4d9f-455d-b26f-8f63f00a1885" path="/var/lib/kubelet/pods/2f052551-4d9f-455d-b26f-8f63f00a1885/volumes"
Nov 02 22:29:30.983084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983053    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"]
Nov 02 22:29:30.983513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983499    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5429792d-7e77-4dff-85e5-60ad855937f8" containerName="revoke-keys"
Nov 02 22:29:30.983513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983514    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5429792d-7e77-4dff-85e5-60ad855937f8" containerName="revoke-keys"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983524    4528 cpu_manager.go:414] "containerMap: removing container" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983529    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983538    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983544    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983552    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7e4750f1-535c-43c2-a353-db31322f9df0" containerName="revoke-keys"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983556    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4750f1-535c-43c2-a353-db31322f9df0" containerName="revoke-keys"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983565    4528 cpu_manager.go:414] "containerMap: removing container" podUID="34a59620-b8fe-456f-afb8-eb95c0de11ac" containerName="revoke-keys"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983569    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a59620-b8fe-456f-afb8-eb95c0de11ac" containerName="revoke-keys"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983583    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983587    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983597    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main"
Nov 02 22:29:30.983603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983601    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983611    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b092804d-6683-4b47-9d26-ad6d9d2df062" containerName="payload-processing"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983616    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b092804d-6683-4b47-9d26-ad6d9d2df062" containerName="payload-processing"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983624    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f7bc743a-7ada-48a6-95be-d6adfedbac23" containerName="payload-processing"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983630    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bc743a-7ada-48a6-95be-d6adfedbac23" containerName="payload-processing"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983641    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2f052551-4d9f-455d-b26f-8f63f00a1885" containerName="payload-pre-processing"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983646    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f052551-4d9f-455d-b26f-8f63f00a1885" containerName="payload-pre-processing"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983708    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983715    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4750f1-535c-43c2-a353-db31322f9df0" containerName="revoke-keys"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983722    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a59620-b8fe-456f-afb8-eb95c0de11ac" containerName="revoke-keys"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983729    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21efa86-2527-440b-acd0-657466f62f1e" containerName="main"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983738    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b092804d-6683-4b47-9d26-ad6d9d2df062" containerName="payload-processing"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983744    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bc743a-7ada-48a6-95be-d6adfedbac23" containerName="payload-processing"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983752    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f052551-4d9f-455d-b26f-8f63f00a1885" containerName="payload-pre-processing"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983757    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5429792d-7e77-4dff-85e5-60ad855937f8" containerName="revoke-keys"
Nov 02 22:29:30.984014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.983762    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="182772ef-d7fb-465f-94f6-20e4fc63e7f5" containerName="main"
Nov 02 22:29:30.988675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:30.988659    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.093069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.093030    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:29:31.093274 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.093128    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:29:31.093274 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.093136    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:29:31.093274 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.093130    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:29:31.100527 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.100502    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"]
Nov 02 22:29:31.152419 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.152390    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152b5df3-ghnj9\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.152564 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.152447    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7ks\" (UniqueName: \"kubernetes.io/projected/24a3eccb-6bf3-4188-8644-d67d36677817-kube-api-access-vx7ks\") pod \"maas-api-revoke-sub-152b5df3-ghnj9\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.152564 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.152536    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152b5df3-ghnj9\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.253317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.253229    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152b5df3-ghnj9\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.253317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.253310    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7ks\" (UniqueName: \"kubernetes.io/projected/24a3eccb-6bf3-4188-8644-d67d36677817-kube-api-access-vx7ks\") pod \"maas-api-revoke-sub-152b5df3-ghnj9\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.253564 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.253345    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152b5df3-ghnj9\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.253890 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.253864    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152b5df3-ghnj9\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.255711 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.255680    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152b5df3-ghnj9\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.284966 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.284935    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw" podUID="50ad4b49-ac5e-41cf-902f-09c529d3b6dc" containerName="payload-processing" containerID="cri-o://a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b" gracePeriod=25
Nov 02 22:29:31.310075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.310044    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7ks\" (UniqueName: \"kubernetes.io/projected/24a3eccb-6bf3-4188-8644-d67d36677817-kube-api-access-vx7ks\") pod \"maas-api-revoke-sub-152b5df3-ghnj9\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.381957 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.381907    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t" podUID="887ce760-4401-437a-a9b5-e1733da74fd7" containerName="payload-pre-processing" containerID="cri-o://153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329" gracePeriod=25
Nov 02 22:29:31.601672 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.601641    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:31.970407 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:29:31.970360    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a3eccb_6bf3_4188_8644_d67d36677817.slice/crio-d7cfc082c23cda94b803a83c9c27de2d7f93d7b006655195cec373da86c0062d WatchSource:0}: Error finding container d7cfc082c23cda94b803a83c9c27de2d7f93d7b006655195cec373da86c0062d: Status 404 returned error can't find the container with id d7cfc082c23cda94b803a83c9c27de2d7f93d7b006655195cec373da86c0062d
Nov 02 22:29:31.973044 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:31.973015    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"]
Nov 02 22:29:32.670282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:32.670202    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9" event={"ID":"24a3eccb-6bf3-4188-8644-d67d36677817","Type":"ContainerStarted","Data":"2c440de76c019a994a3135345c74ddad93a59e6e38025ae57d44db9bf1f481ff"}
Nov 02 22:29:32.670282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:32.670264    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9" event={"ID":"24a3eccb-6bf3-4188-8644-d67d36677817","Type":"ContainerStarted","Data":"d7cfc082c23cda94b803a83c9c27de2d7f93d7b006655195cec373da86c0062d"}
Nov 02 22:29:32.786134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:32.786067    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9" podStartSLOduration=2.786052012 podStartE2EDuration="2.786052012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:32.783005647 +0000 UTC m=+4860.046541964" watchObservedRunningTime="2026-11-02 22:29:32.786052012 +0000 UTC m=+4860.049588329"
Nov 02 22:29:33.678098 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:33.678065    4528 generic.go:356] "Generic (PLEG): container finished" podID="24a3eccb-6bf3-4188-8644-d67d36677817" containerID="2c440de76c019a994a3135345c74ddad93a59e6e38025ae57d44db9bf1f481ff" exitCode=0
Nov 02 22:29:33.678098 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:33.678101    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9" event={"ID":"24a3eccb-6bf3-4188-8644-d67d36677817","Type":"ContainerDied","Data":"2c440de76c019a994a3135345c74ddad93a59e6e38025ae57d44db9bf1f481ff"}
Nov 02 22:29:34.174474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:34.174435    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t" podUID="887ce760-4401-437a-a9b5-e1733da74fd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:29:34.174651 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:34.174487    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw" podUID="50ad4b49-ac5e-41cf-902f-09c529d3b6dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:29:34.243387 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:34.243350    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7" podUID="e9e8e764-309c-4788-8268-b27f2dc8f290" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:29:34.851917 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:34.851894    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:35.010657 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.010575    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-xks-ca\") pod \"24a3eccb-6bf3-4188-8644-d67d36677817\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") "
Nov 02 22:29:35.010831 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.010712    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-openshift-ca\") pod \"24a3eccb-6bf3-4188-8644-d67d36677817\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") "
Nov 02 22:29:35.010831 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.010753    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a3eccb-6bf3-4188-8644-d67d36677817-kube-api-access-vx7ks\" (UniqueName: \"kubernetes.io/projected/24a3eccb-6bf3-4188-8644-d67d36677817-kube-api-access-vx7ks\") pod \"24a3eccb-6bf3-4188-8644-d67d36677817\" (UID: \"24a3eccb-6bf3-4188-8644-d67d36677817\") "
Nov 02 22:29:35.011568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.011539    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-openshift-ca" pod "24a3eccb-6bf3-4188-8644-d67d36677817" (UID: "24a3eccb-6bf3-4188-8644-d67d36677817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:35.013214 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.013192    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-xks-ca" pod "24a3eccb-6bf3-4188-8644-d67d36677817" (UID: "24a3eccb-6bf3-4188-8644-d67d36677817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:35.013341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.013328    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a3eccb-6bf3-4188-8644-d67d36677817-kube-api-access-vx7ks" pod "24a3eccb-6bf3-4188-8644-d67d36677817" (UID: "24a3eccb-6bf3-4188-8644-d67d36677817"). InnerVolumeSpecName "kube-api-access-vx7ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:35.112566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.112532    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:35.112566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.112563    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx7ks\" (UniqueName: \"kubernetes.io/projected/24a3eccb-6bf3-4188-8644-d67d36677817-kube-api-access-vx7ks\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:35.112566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.112573    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24a3eccb-6bf3-4188-8644-d67d36677817-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:35.691312 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.691284    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"
Nov 02 22:29:35.691491 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.691281    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9" event={"ID":"24a3eccb-6bf3-4188-8644-d67d36677817","Type":"ContainerDied","Data":"d7cfc082c23cda94b803a83c9c27de2d7f93d7b006655195cec373da86c0062d"}
Nov 02 22:29:35.691491 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:35.691437    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7cfc082c23cda94b803a83c9c27de2d7f93d7b006655195cec373da86c0062d"
Nov 02 22:29:40.814025 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.813988    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"]
Nov 02 22:29:40.814525 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.814509    4528 cpu_manager.go:414] "containerMap: removing container" podUID="24a3eccb-6bf3-4188-8644-d67d36677817" containerName="revoke-keys"
Nov 02 22:29:40.814575 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.814526    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a3eccb-6bf3-4188-8644-d67d36677817" containerName="revoke-keys"
Nov 02 22:29:40.814624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.814615    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33afd9e-0e68-4231-910d-28348af5ab93" containerName="istio-proxy"
Nov 02 22:29:40.814659 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.814625    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a3eccb-6bf3-4188-8644-d67d36677817" containerName="revoke-keys"
Nov 02 22:29:40.819744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.819725    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:40.875375 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.875340    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"]
Nov 02 22:29:40.882530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.882497    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46641806-qn7xj\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:40.882682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.882569    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9rg\" (UniqueName: \"kubernetes.io/projected/321bace5-9613-4ad1-8b99-a2083bc2daf4-kube-api-access-lx9rg\") pod \"maas-api-revoke-sub-46641806-qn7xj\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:40.882728 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.882710    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46641806-qn7xj\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:40.922163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.922129    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:29:40.922328 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.922203    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:29:40.922328 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.922281    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:29:40.922588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.922572    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:29:40.984080 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.984041    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46641806-qn7xj\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:40.984318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.984140    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9rg\" (UniqueName: \"kubernetes.io/projected/321bace5-9613-4ad1-8b99-a2083bc2daf4-kube-api-access-lx9rg\") pod \"maas-api-revoke-sub-46641806-qn7xj\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:40.984318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.984299    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46641806-qn7xj\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:40.984986 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.984963    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46641806-qn7xj\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:40.986814 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:40.986798    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46641806-qn7xj\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:41.041138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:41.041094    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9rg\" (UniqueName: \"kubernetes.io/projected/321bace5-9613-4ad1-8b99-a2083bc2daf4-kube-api-access-lx9rg\") pod \"maas-api-revoke-sub-46641806-qn7xj\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:41.139643 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:41.139574    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:41.510097 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:41.510036    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"]
Nov 02 22:29:41.735535 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:41.735506    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj" event={"ID":"321bace5-9613-4ad1-8b99-a2083bc2daf4","Type":"ContainerStarted","Data":"14184a1beb56d154555eab0db466a301a88c21dd7a8df72524a4a4b93b4712a4"}
Nov 02 22:29:42.745926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:42.745830    4528 generic.go:356] "Generic (PLEG): container finished" podID="321bace5-9613-4ad1-8b99-a2083bc2daf4" containerID="a6f359ba5bfcd71cb563895f735d56c642cca82dc51d1191262b434b052f237c" exitCode=0
Nov 02 22:29:42.746753 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:42.745918    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj" event={"ID":"321bace5-9613-4ad1-8b99-a2083bc2daf4","Type":"ContainerDied","Data":"a6f359ba5bfcd71cb563895f735d56c642cca82dc51d1191262b434b052f237c"}
Nov 02 22:29:42.853066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:42.853032    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"]
Nov 02 22:29:43.336379 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.336318    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 22:29:43.412373 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.412336    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e8e764-309c-4788-8268-b27f2dc8f290-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e8e764-309c-4788-8268-b27f2dc8f290-plugins-config-volume\") pod \"e9e8e764-309c-4788-8268-b27f2dc8f290\" (UID: \"e9e8e764-309c-4788-8268-b27f2dc8f290\") "
Nov 02 22:29:43.412568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.412387    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e8e764-309c-4788-8268-b27f2dc8f290-kube-api-access-twvcp\" (UniqueName: \"kubernetes.io/projected/e9e8e764-309c-4788-8268-b27f2dc8f290-kube-api-access-twvcp\") pod \"e9e8e764-309c-4788-8268-b27f2dc8f290\" (UID: \"e9e8e764-309c-4788-8268-b27f2dc8f290\") "
Nov 02 22:29:43.412683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.412660    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e8e764-309c-4788-8268-b27f2dc8f290-plugins-config-volume" pod "e9e8e764-309c-4788-8268-b27f2dc8f290" (UID: "e9e8e764-309c-4788-8268-b27f2dc8f290"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:43.412807 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.412789    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e8e764-309c-4788-8268-b27f2dc8f290-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:43.414535 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.414512    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e8e764-309c-4788-8268-b27f2dc8f290-kube-api-access-twvcp" pod "e9e8e764-309c-4788-8268-b27f2dc8f290" (UID: "e9e8e764-309c-4788-8268-b27f2dc8f290"). InnerVolumeSpecName "kube-api-access-twvcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:43.514311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.514274    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twvcp\" (UniqueName: \"kubernetes.io/projected/e9e8e764-309c-4788-8268-b27f2dc8f290-kube-api-access-twvcp\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:43.753006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.752849    4528 generic.go:356] "Generic (PLEG): container finished" podID="e9e8e764-309c-4788-8268-b27f2dc8f290" containerID="e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c" exitCode=0
Nov 02 22:29:43.753006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.752894    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7" event={"ID":"e9e8e764-309c-4788-8268-b27f2dc8f290","Type":"ContainerDied","Data":"e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c"}
Nov 02 22:29:43.753006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.752959    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7" event={"ID":"e9e8e764-309c-4788-8268-b27f2dc8f290","Type":"ContainerDied","Data":"7e636f4d436467d1865a2a6b92f90ac054bf18a2235229a7ddd923f7b805b33b"}
Nov 02 22:29:43.753006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.752901    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"
Nov 02 22:29:43.753933 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.752986    4528 scope.go:122] "RemoveContainer" containerID="e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c"
Nov 02 22:29:43.767211 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.767186    4528 scope.go:122] "RemoveContainer" containerID="e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c"
Nov 02 22:29:43.767593 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:43.767575    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c\": container with ID starting with e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c not found: ID does not exist" containerID="e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c"
Nov 02 22:29:43.767672 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.767602    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c"} err="failed to get container status \"e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c\": rpc error: code = NotFound desc = could not find container \"e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c\": container with ID starting with e2b5af852f427a0e33f9a16d3176cb231a53b2e75af326b8900308aca5f0043c not found: ID does not exist"
Nov 02 22:29:43.881702 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.881668    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:43.918367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.918336    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"]
Nov 02 22:29:43.918367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.918339    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-xks-ca\") pod \"321bace5-9613-4ad1-8b99-a2083bc2daf4\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") "
Nov 02 22:29:43.918589 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.918564    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321bace5-9613-4ad1-8b99-a2083bc2daf4-kube-api-access-lx9rg\" (UniqueName: \"kubernetes.io/projected/321bace5-9613-4ad1-8b99-a2083bc2daf4-kube-api-access-lx9rg\") pod \"321bace5-9613-4ad1-8b99-a2083bc2daf4\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") "
Nov 02 22:29:43.918649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.918633    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-openshift-ca\") pod \"321bace5-9613-4ad1-8b99-a2083bc2daf4\" (UID: \"321bace5-9613-4ad1-8b99-a2083bc2daf4\") "
Nov 02 22:29:43.919024 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.918997    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-openshift-ca" pod "321bace5-9613-4ad1-8b99-a2083bc2daf4" (UID: "321bace5-9613-4ad1-8b99-a2083bc2daf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:43.922334 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.920595    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-xks-ca" pod "321bace5-9613-4ad1-8b99-a2083bc2daf4" (UID: "321bace5-9613-4ad1-8b99-a2083bc2daf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:43.923857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.923782    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321bace5-9613-4ad1-8b99-a2083bc2daf4-kube-api-access-lx9rg" pod "321bace5-9613-4ad1-8b99-a2083bc2daf4" (UID: "321bace5-9613-4ad1-8b99-a2083bc2daf4"). InnerVolumeSpecName "kube-api-access-lx9rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:43.927076 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:43.927054    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6fa533-7686998999-5zrz7"]
Nov 02 22:29:44.020069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:44.020035    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9rg\" (UniqueName: \"kubernetes.io/projected/321bace5-9613-4ad1-8b99-a2083bc2daf4-kube-api-access-lx9rg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:44.020069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:44.020066    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:44.020308 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:44.020080    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321bace5-9613-4ad1-8b99-a2083bc2daf4-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:44.114816 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:44.114783    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:29:44.174239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:44.174201    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t" podUID="887ce760-4401-437a-a9b5-e1733da74fd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:29:44.174379 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:44.174278    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw" podUID="50ad4b49-ac5e-41cf-902f-09c529d3b6dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:29:44.760690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:44.760662    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"
Nov 02 22:29:44.760690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:44.760671    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj" event={"ID":"321bace5-9613-4ad1-8b99-a2083bc2daf4","Type":"ContainerDied","Data":"14184a1beb56d154555eab0db466a301a88c21dd7a8df72524a4a4b93b4712a4"}
Nov 02 22:29:44.761186 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:44.760718    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14184a1beb56d154555eab0db466a301a88c21dd7a8df72524a4a4b93b4712a4"
Nov 02 22:29:44.911818 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:44.911785    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e8e764-309c-4788-8268-b27f2dc8f290" path="/var/lib/kubelet/pods/e9e8e764-309c-4788-8268-b27f2dc8f290/volumes"
Nov 02 22:29:45.212225 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:45.212140    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"]
Nov 02 22:29:46.410757 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:46.410728    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:29:49.189916 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:49.189868    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"]
Nov 02 22:29:49.190414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:49.190063    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy" containerID="cri-o://008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8" gracePeriod=30
Nov 02 22:29:50.351475 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:50.351441    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:29:50.351935 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:50.351502    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:29:51.572979 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.572930    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 22:29:51.613313 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.613276    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-kube-api-access-9zzfj\" (UniqueName: \"kubernetes.io/projected/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-kube-api-access-9zzfj\") pod \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\" (UID: \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\") "
Nov 02 22:29:51.613506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.613339    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-plugins-config-volume\") pod \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\" (UID: \"50ad4b49-ac5e-41cf-902f-09c529d3b6dc\") "
Nov 02 22:29:51.614069 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.614014    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-plugins-config-volume" pod "50ad4b49-ac5e-41cf-902f-09c529d3b6dc" (UID: "50ad4b49-ac5e-41cf-902f-09c529d3b6dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:51.616790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.616760    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-kube-api-access-9zzfj" pod "50ad4b49-ac5e-41cf-902f-09c529d3b6dc" (UID: "50ad4b49-ac5e-41cf-902f-09c529d3b6dc"). InnerVolumeSpecName "kube-api-access-9zzfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:51.644888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.644868    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 22:29:51.715949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.715146    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887ce760-4401-437a-a9b5-e1733da74fd7-kube-api-access-4cskg\" (UniqueName: \"kubernetes.io/projected/887ce760-4401-437a-a9b5-e1733da74fd7-kube-api-access-4cskg\") pod \"887ce760-4401-437a-a9b5-e1733da74fd7\" (UID: \"887ce760-4401-437a-a9b5-e1733da74fd7\") "
Nov 02 22:29:51.715949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.715234    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887ce760-4401-437a-a9b5-e1733da74fd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887ce760-4401-437a-a9b5-e1733da74fd7-plugins-config-volume\") pod \"887ce760-4401-437a-a9b5-e1733da74fd7\" (UID: \"887ce760-4401-437a-a9b5-e1733da74fd7\") "
Nov 02 22:29:51.715949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.715661    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zzfj\" (UniqueName: \"kubernetes.io/projected/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-kube-api-access-9zzfj\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:51.715949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.715683    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad4b49-ac5e-41cf-902f-09c529d3b6dc-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:51.716586 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.716554    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887ce760-4401-437a-a9b5-e1733da74fd7-plugins-config-volume" pod "887ce760-4401-437a-a9b5-e1733da74fd7" (UID: "887ce760-4401-437a-a9b5-e1733da74fd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:51.718050 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.718026    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887ce760-4401-437a-a9b5-e1733da74fd7-kube-api-access-4cskg" pod "887ce760-4401-437a-a9b5-e1733da74fd7" (UID: "887ce760-4401-437a-a9b5-e1733da74fd7"). InnerVolumeSpecName "kube-api-access-4cskg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:51.813320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.813237    4528 generic.go:356] "Generic (PLEG): container finished" podID="887ce760-4401-437a-a9b5-e1733da74fd7" containerID="153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329" exitCode=0
Nov 02 22:29:51.813320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.813304    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"
Nov 02 22:29:51.813570 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.813315    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t" event={"ID":"887ce760-4401-437a-a9b5-e1733da74fd7","Type":"ContainerDied","Data":"153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329"}
Nov 02 22:29:51.813570 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.813404    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t" event={"ID":"887ce760-4401-437a-a9b5-e1733da74fd7","Type":"ContainerDied","Data":"5a172c5440c24a1231cf673ee48c63bb811cea18a38e1d50ff10f2d773f7954f"}
Nov 02 22:29:51.813570 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.813437    4528 scope.go:122] "RemoveContainer" containerID="153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329"
Nov 02 22:29:51.815433 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.815405    4528 generic.go:356] "Generic (PLEG): container finished" podID="50ad4b49-ac5e-41cf-902f-09c529d3b6dc" containerID="a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b" exitCode=0
Nov 02 22:29:51.815578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.815477    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw" event={"ID":"50ad4b49-ac5e-41cf-902f-09c529d3b6dc","Type":"ContainerDied","Data":"a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b"}
Nov 02 22:29:51.815578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.815524    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw" event={"ID":"50ad4b49-ac5e-41cf-902f-09c529d3b6dc","Type":"ContainerDied","Data":"49970b45f5324df5ac119a63ff15bee40c4fe88f543e132e390caa8f42a1ce12"}
Nov 02 22:29:51.815578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.815566    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"
Nov 02 22:29:51.816662 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.816638    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cskg\" (UniqueName: \"kubernetes.io/projected/887ce760-4401-437a-a9b5-e1733da74fd7-kube-api-access-4cskg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:51.816803 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.816665    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887ce760-4401-437a-a9b5-e1733da74fd7-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:51.826775 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.826750    4528 scope.go:122] "RemoveContainer" containerID="153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329"
Nov 02 22:29:51.827159 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:51.827134    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329\": container with ID starting with 153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329 not found: ID does not exist" containerID="153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329"
Nov 02 22:29:51.827253 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.827172    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329"} err="failed to get container status \"153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329\": rpc error: code = NotFound desc = could not find container \"153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329\": container with ID starting with 153c8a1bfc9c930bff9800a99f49fbd65e243dfc3346568890b93afe074dd329 not found: ID does not exist"
Nov 02 22:29:51.827253 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.827199    4528 scope.go:122] "RemoveContainer" containerID="a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b"
Nov 02 22:29:51.837346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.837329    4528 scope.go:122] "RemoveContainer" containerID="a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b"
Nov 02 22:29:51.837631 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:51.837611    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b\": container with ID starting with a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b not found: ID does not exist" containerID="a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b"
Nov 02 22:29:51.837698 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.837638    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b"} err="failed to get container status \"a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b\": rpc error: code = NotFound desc = could not find container \"a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b\": container with ID starting with a7f9dd22e8b8301cee2fd3f5c41e72e498c0cede0b4f42576d75abacf63fc01b not found: ID does not exist"
Nov 02 22:29:51.983928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.983895    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"]
Nov 02 22:29:51.986788 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:51.986757    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ca7f-6bf9c686c-jdhzw"]
Nov 02 22:29:52.097699 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:52.097624    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"]
Nov 02 22:29:52.101919 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:52.101890    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ca7f-bd86588f9-dpr5t"]
Nov 02 22:29:52.913471 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:52.913436    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ad4b49-ac5e-41cf-902f-09c529d3b6dc" path="/var/lib/kubelet/pods/50ad4b49-ac5e-41cf-902f-09c529d3b6dc/volumes"
Nov 02 22:29:52.913848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:52.913840    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887ce760-4401-437a-a9b5-e1733da74fd7" path="/var/lib/kubelet/pods/887ce760-4401-437a-a9b5-e1733da74fd7/volumes"
Nov 02 22:29:54.660131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.660093    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 22:29:54.776123 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776088    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-podinfo\") pod \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") "
Nov 02 22:29:54.776344 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776193    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-socket\") pod \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") "
Nov 02 22:29:54.776344 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776242    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-credential-socket\") pod \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") "
Nov 02 22:29:54.776344 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776267    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-token\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-token\") pod \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") "
Nov 02 22:29:54.776344 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776323    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-envoy\") pod \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") "
Nov 02 22:29:54.776538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776349    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-certs\") pod \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") "
Nov 02 22:29:54.776538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776465    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istiod-ca-cert\") pod \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") "
Nov 02 22:29:54.776538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776502    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-data\") pod \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") "
Nov 02 22:29:54.776538 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776504    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-socket" pod "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" (UID: "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:54.776721 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776536    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-kube-api-access-fhwr4\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-kube-api-access-fhwr4\") pod \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\" (UID: \"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a\") "
Nov 02 22:29:54.776721 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776542    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-credential-socket" pod "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" (UID: "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:54.776819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776753    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-certs" pod "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" (UID: "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:54.776819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776783    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istiod-ca-cert" pod "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" (UID: "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:29:54.776921 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.776894    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-data" pod "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" (UID: "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:54.777075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.777048    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:54.777075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.777076    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:54.777297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.777091    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:54.777297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.777104    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:54.777297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.777133    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:54.778888 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.778858    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-podinfo" pod "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" (UID: "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:29:54.779611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.779571    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-envoy" pod "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" (UID: "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:54.779611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.779587    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-kube-api-access-fhwr4" pod "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" (UID: "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a"). InnerVolumeSpecName "kube-api-access-fhwr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:54.779745 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.779616    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-token" pod "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" (UID: "ed01ac5b-9b7a-42b8-8053-d2e0c53e149a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:54.836730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.836683    4528 generic.go:356] "Generic (PLEG): container finished" podID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerID="008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8" exitCode=0
Nov 02 22:29:54.836895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.836746    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"
Nov 02 22:29:54.836895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.836842    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" event={"ID":"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a","Type":"ContainerDied","Data":"008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8"}
Nov 02 22:29:54.836895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.836888    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5" event={"ID":"ed01ac5b-9b7a-42b8-8053-d2e0c53e149a","Type":"ContainerDied","Data":"794c0f7213060b6f8b77ba556988f781665b47943e4a4b66df8b2521c5ce6c5b"}
Nov 02 22:29:54.837101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.836923    4528 scope.go:122] "RemoveContainer" containerID="008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8"
Nov 02 22:29:54.849998 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.849975    4528 scope.go:122] "RemoveContainer" containerID="008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8"
Nov 02 22:29:54.850380 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:54.850354    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8\": container with ID starting with 008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8 not found: ID does not exist" containerID="008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8"
Nov 02 22:29:54.850465 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.850394    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8"} err="failed to get container status \"008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8\": rpc error: code = NotFound desc = could not find container \"008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8\": container with ID starting with 008491a8353e085e0e4cfefc756f576d0c6fdfea716170f0a6c8bce3ab13c9f8 not found: ID does not exist"
Nov 02 22:29:54.877743 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.877670    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:54.877743 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.877698    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:54.877743 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.877715    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhwr4\" (UniqueName: \"kubernetes.io/projected/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-kube-api-access-fhwr4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:54.877743 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:54.877729    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:55.006484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:55.006428    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"]
Nov 02 22:29:55.009373 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:55.009346    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t4dr5"]
Nov 02 22:29:56.912507 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:56.912471    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" path="/var/lib/kubelet/pods/ed01ac5b-9b7a-42b8-8053-d2e0c53e149a/volumes"
Nov 02 22:29:57.218485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218411    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"]
Nov 02 22:29:57.218835 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218824    4528 cpu_manager.go:414] "containerMap: removing container" podUID="887ce760-4401-437a-a9b5-e1733da74fd7" containerName="payload-pre-processing"
Nov 02 22:29:57.218883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218837    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887ce760-4401-437a-a9b5-e1733da74fd7" containerName="payload-pre-processing"
Nov 02 22:29:57.218883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218856    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy"
Nov 02 22:29:57.218883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218861    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy"
Nov 02 22:29:57.218883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218879    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e9e8e764-309c-4788-8268-b27f2dc8f290" containerName="payload-pre-processing"
Nov 02 22:29:57.218883 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218884    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e8e764-309c-4788-8268-b27f2dc8f290" containerName="payload-pre-processing"
Nov 02 22:29:57.219046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218898    4528 cpu_manager.go:414] "containerMap: removing container" podUID="50ad4b49-ac5e-41cf-902f-09c529d3b6dc" containerName="payload-processing"
Nov 02 22:29:57.219046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218903    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ad4b49-ac5e-41cf-902f-09c529d3b6dc" containerName="payload-processing"
Nov 02 22:29:57.219046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218915    4528 cpu_manager.go:414] "containerMap: removing container" podUID="321bace5-9613-4ad1-8b99-a2083bc2daf4" containerName="revoke-keys"
Nov 02 22:29:57.219046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218919    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321bace5-9613-4ad1-8b99-a2083bc2daf4" containerName="revoke-keys"
Nov 02 22:29:57.219046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218976    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ad4b49-ac5e-41cf-902f-09c529d3b6dc" containerName="payload-processing"
Nov 02 22:29:57.219046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218987    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="321bace5-9613-4ad1-8b99-a2083bc2daf4" containerName="revoke-keys"
Nov 02 22:29:57.219046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.218993    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="887ce760-4401-437a-a9b5-e1733da74fd7" containerName="payload-pre-processing"
Nov 02 22:29:57.219046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.219001    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed01ac5b-9b7a-42b8-8053-d2e0c53e149a" containerName="istio-proxy"
Nov 02 22:29:57.219046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.219008    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e8e764-309c-4788-8268-b27f2dc8f290" containerName="payload-pre-processing"
Nov 02 22:29:57.223668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.223652    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.280951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.280905    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"]
Nov 02 22:29:57.305544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.305513    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a9f62f-2ttml\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.305705 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.305642    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shf7b\" (UniqueName: \"kubernetes.io/projected/22a24acc-dc6f-4ef9-b75c-332992234503-kube-api-access-shf7b\") pod \"maas-api-revoke-sub-e0a9f62f-2ttml\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.305766 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.305751    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a9f62f-2ttml\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.326854 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.326824    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:29:57.327029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.326845    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:29:57.327029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.326866    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:29:57.327029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.326871    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:29:57.406262 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.406228    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shf7b\" (UniqueName: \"kubernetes.io/projected/22a24acc-dc6f-4ef9-b75c-332992234503-kube-api-access-shf7b\") pod \"maas-api-revoke-sub-e0a9f62f-2ttml\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.406463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.406308    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a9f62f-2ttml\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.406463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.406349    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a9f62f-2ttml\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.407103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.407077    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a9f62f-2ttml\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.408735 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.408713    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a9f62f-2ttml\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.463239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.463208    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shf7b\" (UniqueName: \"kubernetes.io/projected/22a24acc-dc6f-4ef9-b75c-332992234503-kube-api-access-shf7b\") pod \"maas-api-revoke-sub-e0a9f62f-2ttml\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.536224 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.536192    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:29:57.872969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.872877    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv" podUID="d2bee1ea-ab9d-421f-badf-59117f3de3ac" containerName="main" containerID="cri-o://0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91" gracePeriod=45
Nov 02 22:29:57.909420 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:57.909393    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"]
Nov 02 22:29:57.921090 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:29:57.921065    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a24acc_dc6f_4ef9_b75c_332992234503.slice/crio-0b0945c39b2ac378fb0a6e4abaff6aada655464445deee935011f3b9d2d883eb WatchSource:0}: Error finding container 0b0945c39b2ac378fb0a6e4abaff6aada655464445deee935011f3b9d2d883eb: Status 404 returned error can't find the container with id 0b0945c39b2ac378fb0a6e4abaff6aada655464445deee935011f3b9d2d883eb
Nov 02 22:29:58.115009 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.114984    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:29:58.213884 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.213817    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2bee1ea-ab9d-421f-badf-59117f3de3ac-kube-api-access-fp2zt\" (UniqueName: \"kubernetes.io/projected/d2bee1ea-ab9d-421f-badf-59117f3de3ac-kube-api-access-fp2zt\") pod \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") "
Nov 02 22:29:58.214027 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.213927    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tls-certs\") pod \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") "
Nov 02 22:29:58.214027 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.213947    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-model-cache\") pod \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") "
Nov 02 22:29:58.214027 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.213985    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-home\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-home\") pod \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") "
Nov 02 22:29:58.214170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.214054    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-dshm\") pod \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") "
Nov 02 22:29:58.214170 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.214074    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tmp-dir\") pod \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\" (UID: \"d2bee1ea-ab9d-421f-badf-59117f3de3ac\") "
Nov 02 22:29:58.214278 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.214233    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-model-cache" pod "d2bee1ea-ab9d-421f-badf-59117f3de3ac" (UID: "d2bee1ea-ab9d-421f-badf-59117f3de3ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:58.214340 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.214324    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-home" pod "d2bee1ea-ab9d-421f-badf-59117f3de3ac" (UID: "d2bee1ea-ab9d-421f-badf-59117f3de3ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:58.214441 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.214417    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tmp-dir" pod "d2bee1ea-ab9d-421f-badf-59117f3de3ac" (UID: "d2bee1ea-ab9d-421f-badf-59117f3de3ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:58.214573 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.214557    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:58.214630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.214579    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:58.214630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.214593    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:58.216131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.216093    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-dshm" pod "d2bee1ea-ab9d-421f-badf-59117f3de3ac" (UID: "d2bee1ea-ab9d-421f-badf-59117f3de3ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:29:58.216430 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.216415    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tls-certs" pod "d2bee1ea-ab9d-421f-badf-59117f3de3ac" (UID: "d2bee1ea-ab9d-421f-badf-59117f3de3ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:29:58.216474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.216447    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bee1ea-ab9d-421f-badf-59117f3de3ac-kube-api-access-fp2zt" pod "d2bee1ea-ab9d-421f-badf-59117f3de3ac" (UID: "d2bee1ea-ab9d-421f-badf-59117f3de3ac"). InnerVolumeSpecName "kube-api-access-fp2zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:29:58.315585 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.315556    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bee1ea-ab9d-421f-badf-59117f3de3ac-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:58.315585 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.315581    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2zt\" (UniqueName: \"kubernetes.io/projected/d2bee1ea-ab9d-421f-badf-59117f3de3ac-kube-api-access-fp2zt\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:58.315585 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.315591    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bee1ea-ab9d-421f-badf-59117f3de3ac-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:29:58.871250 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.871210    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml" event={"ID":"22a24acc-dc6f-4ef9-b75c-332992234503","Type":"ContainerStarted","Data":"2c694fba54f5d6e11a331cb3f31d85e4d77a705c864ed32274a6e268cfea915e"}
Nov 02 22:29:58.871449 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.871281    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml" event={"ID":"22a24acc-dc6f-4ef9-b75c-332992234503","Type":"ContainerStarted","Data":"0b0945c39b2ac378fb0a6e4abaff6aada655464445deee935011f3b9d2d883eb"}
Nov 02 22:29:58.873987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.873956    4528 generic.go:356] "Generic (PLEG): container finished" podID="d2bee1ea-ab9d-421f-badf-59117f3de3ac" containerID="0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91" exitCode=0
Nov 02 22:29:58.874147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.873998    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv" event={"ID":"d2bee1ea-ab9d-421f-badf-59117f3de3ac","Type":"ContainerDied","Data":"0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91"}
Nov 02 22:29:58.874147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.874018    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"
Nov 02 22:29:58.874147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.874038    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv" event={"ID":"d2bee1ea-ab9d-421f-badf-59117f3de3ac","Type":"ContainerDied","Data":"51b3fd9c9aef8e95bc75b4f2e5a774ca07ddae26e5654ce4a8c3f4c070f7683d"}
Nov 02 22:29:58.874147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.874060    4528 scope.go:122] "RemoveContainer" containerID="0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91"
Nov 02 22:29:58.884328 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.884307    4528 scope.go:122] "RemoveContainer" containerID="0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91"
Nov 02 22:29:58.884626 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:29:58.884602    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91\": container with ID starting with 0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91 not found: ID does not exist" containerID="0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91"
Nov 02 22:29:58.884679 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.884639    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91"} err="failed to get container status \"0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91\": rpc error: code = NotFound desc = could not find container \"0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91\": container with ID starting with 0b0a1423fcf09d168eb410a0f42cd756f5ac68ec3788e4fef782d563e90c1f91 not found: ID does not exist"
Nov 02 22:29:58.984678 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:58.984622    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml" podStartSLOduration=1.984609994 podStartE2EDuration="1.984609994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:58.983897785 +0000 UTC m=+4886.247434098" watchObservedRunningTime="2026-11-02 22:29:58.984609994 +0000 UTC m=+4886.248146347"
Nov 02 22:29:59.096179 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:59.096147    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"]
Nov 02 22:29:59.101234 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:59.101212    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/rate-test-model-6e535d-kserve-6759d75857-fgbvv"]
Nov 02 22:29:59.881428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:59.881394    4528 generic.go:356] "Generic (PLEG): container finished" podID="22a24acc-dc6f-4ef9-b75c-332992234503" containerID="2c694fba54f5d6e11a331cb3f31d85e4d77a705c864ed32274a6e268cfea915e" exitCode=0
Nov 02 22:29:59.881610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:29:59.881478    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml" event={"ID":"22a24acc-dc6f-4ef9-b75c-332992234503","Type":"ContainerDied","Data":"2c694fba54f5d6e11a331cb3f31d85e4d77a705c864ed32274a6e268cfea915e"}
Nov 02 22:30:00.163528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.163440    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"]
Nov 02 22:30:00.164143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.164122    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d2bee1ea-ab9d-421f-badf-59117f3de3ac" containerName="main"
Nov 02 22:30:00.164199 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.164146    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2bee1ea-ab9d-421f-badf-59117f3de3ac" containerName="main"
Nov 02 22:30:00.164279 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.164267    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2bee1ea-ab9d-421f-badf-59117f3de3ac" containerName="main"
Nov 02 22:30:00.167537 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.167513    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"]
Nov 02 22:30:00.167668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.167628    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:00.173045 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.173015    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:00.176371 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.176350    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"]
Nov 02 22:30:00.181887 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.181866    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"]
Nov 02 22:30:00.182012 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.181984    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:00.185809 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.185791    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:00.201721 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.201696    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"]
Nov 02 22:30:00.205981 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.205962    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:00.231232 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.231204    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main" containerID="cri-o://a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe" gracePeriod=45
Nov 02 22:30:00.234601 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.234574    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991fa00a-364a-4544-a478-66e3eac54f46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5\" (UID: \"991fa00a-364a-4544-a478-66e3eac54f46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:00.234718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.234610    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mnr\" (UniqueName: \"kubernetes.io/projected/991fa00a-364a-4544-a478-66e3eac54f46-kube-api-access-v7mnr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5\" (UID: \"991fa00a-364a-4544-a478-66e3eac54f46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:00.234718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.234702    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921a58d-7bdc-4793-bf83-d53ffb571b0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v\" (UID: \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:00.234847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.234723    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2djfl\" (UniqueName: \"kubernetes.io/projected/d921a58d-7bdc-4793-bf83-d53ffb571b0f-kube-api-access-2djfl\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v\" (UID: \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:00.234847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.234774    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr4cm\" (UniqueName: \"kubernetes.io/projected/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-kube-api-access-dr4cm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl\" (UID: \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:00.234847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.234834    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl\" (UID: \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:00.234973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.234853    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc251060-916a-4354-9ad9-35a9fd77fada-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv\" (UID: \"cc251060-916a-4354-9ad9-35a9fd77fada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:00.234973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.234920    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9vp2\" (UniqueName: \"kubernetes.io/projected/cc251060-916a-4354-9ad9-35a9fd77fada-kube-api-access-n9vp2\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv\" (UID: \"cc251060-916a-4354-9ad9-35a9fd77fada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:00.250074 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.250051    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"]
Nov 02 22:30:00.267878 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.267615    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"]
Nov 02 22:30:00.273735 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.273688    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"]
Nov 02 22:30:00.275663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.275635    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"]
Nov 02 22:30:00.300042 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.299997    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"]
Nov 02 22:30:00.336349 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336318    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921a58d-7bdc-4793-bf83-d53ffb571b0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v\" (UID: \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:00.336547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336355    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2djfl\" (UniqueName: \"kubernetes.io/projected/d921a58d-7bdc-4793-bf83-d53ffb571b0f-kube-api-access-2djfl\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v\" (UID: \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:00.336547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336380    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr4cm\" (UniqueName: \"kubernetes.io/projected/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-kube-api-access-dr4cm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl\" (UID: \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:00.336547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336411    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xck7p\" (UniqueName: \"kubernetes.io/projected/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-kube-api-access-xck7p\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp\" (UID: \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:00.336547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336492    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl\" (UID: \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:00.336811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336567    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc251060-916a-4354-9ad9-35a9fd77fada-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv\" (UID: \"cc251060-916a-4354-9ad9-35a9fd77fada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:00.336811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336609    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp\" (UID: \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:00.336811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336653    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9vp2\" (UniqueName: \"kubernetes.io/projected/cc251060-916a-4354-9ad9-35a9fd77fada-kube-api-access-n9vp2\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv\" (UID: \"cc251060-916a-4354-9ad9-35a9fd77fada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:00.336811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336695    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991fa00a-364a-4544-a478-66e3eac54f46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5\" (UID: \"991fa00a-364a-4544-a478-66e3eac54f46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:00.336811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.336755    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mnr\" (UniqueName: \"kubernetes.io/projected/991fa00a-364a-4544-a478-66e3eac54f46-kube-api-access-v7mnr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5\" (UID: \"991fa00a-364a-4544-a478-66e3eac54f46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:00.337171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.337140    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921a58d-7bdc-4793-bf83-d53ffb571b0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v\" (UID: \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:00.337241 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.337219    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl\" (UID: \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:00.337303 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.337289    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc251060-916a-4354-9ad9-35a9fd77fada-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv\" (UID: \"cc251060-916a-4354-9ad9-35a9fd77fada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:00.337532 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.337514    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991fa00a-364a-4544-a478-66e3eac54f46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5\" (UID: \"991fa00a-364a-4544-a478-66e3eac54f46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:00.394298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.394269    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2djfl\" (UniqueName: \"kubernetes.io/projected/d921a58d-7bdc-4793-bf83-d53ffb571b0f-kube-api-access-2djfl\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v\" (UID: \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:00.394503 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.394453    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr4cm\" (UniqueName: \"kubernetes.io/projected/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-kube-api-access-dr4cm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl\" (UID: \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:00.394624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.394601    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9vp2\" (UniqueName: \"kubernetes.io/projected/cc251060-916a-4354-9ad9-35a9fd77fada-kube-api-access-n9vp2\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv\" (UID: \"cc251060-916a-4354-9ad9-35a9fd77fada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:00.394736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.394721    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mnr\" (UniqueName: \"kubernetes.io/projected/991fa00a-364a-4544-a478-66e3eac54f46-kube-api-access-v7mnr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5\" (UID: \"991fa00a-364a-4544-a478-66e3eac54f46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:00.438454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.438376    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp\" (UID: \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:00.438615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.438545    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xck7p\" (UniqueName: \"kubernetes.io/projected/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-kube-api-access-xck7p\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp\" (UID: \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:00.439020 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.438993    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp\" (UID: \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:00.490431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.490403    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:00.495340 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.495313    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xck7p\" (UniqueName: \"kubernetes.io/projected/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-kube-api-access-xck7p\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp\" (UID: \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:00.497431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.497413    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 22:30:00.498423 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.498406    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:00.507869 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.507845    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:00.517158 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.517135    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:00.522144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.522103    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:00.642194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.641605    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-kube-api-access-5sc5p\" (UniqueName: \"kubernetes.io/projected/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-kube-api-access-5sc5p\") pod \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") "
Nov 02 22:30:00.642194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.641661    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tmp-dir\") pod \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") "
Nov 02 22:30:00.642194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.641754    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-model-cache\") pod \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") "
Nov 02 22:30:00.642194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.641805    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-dshm\") pod \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") "
Nov 02 22:30:00.642194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.641853    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tls-certs\" (UniqueName: \"kubernetes.io/secret/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tls-certs\") pod \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") "
Nov 02 22:30:00.642194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.641900    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-home\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-home\") pod \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\" (UID: \"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612\") "
Nov 02 22:30:00.642989 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.642916    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-model-cache" pod "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" (UID: "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:30:00.643251 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.643193    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tmp-dir" pod "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" (UID: "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:30:00.643251 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.643210    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-home" pod "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" (UID: "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:30:00.643991 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.643964    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-kube-api-access-5sc5p" pod "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" (UID: "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612"). InnerVolumeSpecName "kube-api-access-5sc5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:30:00.645056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.645035    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tls-certs" pod "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" (UID: "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:30:00.645347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.645332    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-dshm" pod "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" (UID: "57c65c87-4bcb-4a5c-b4d1-396f2a7f3612"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:30:00.743171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.743079    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sc5p\" (UniqueName: \"kubernetes.io/projected/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-kube-api-access-5sc5p\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:00.743171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.743122    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:00.743171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.743136    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:00.743171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.743147    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:00.743171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.743160    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:00.743171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.743172    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:00.867658 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:30:00.867622    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cff92e_d2fa_4e9d_8275_1d9e401af54c.slice/crio-6ef6d9d3678be6a1e4eeb659e65e46926579335dbbfc3c49f7623a3e7b059b02 WatchSource:0}: Error finding container 6ef6d9d3678be6a1e4eeb659e65e46926579335dbbfc3c49f7623a3e7b059b02: Status 404 returned error can't find the container with id 6ef6d9d3678be6a1e4eeb659e65e46926579335dbbfc3c49f7623a3e7b059b02
Nov 02 22:30:00.870805 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.870782    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"]
Nov 02 22:30:00.893064 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.893031    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" event={"ID":"a8cff92e-d2fa-4e9d-8275-1d9e401af54c","Type":"ContainerStarted","Data":"6ef6d9d3678be6a1e4eeb659e65e46926579335dbbfc3c49f7623a3e7b059b02"}
Nov 02 22:30:00.894394 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.894372    4528 generic.go:356] "Generic (PLEG): container finished" podID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerID="a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe" exitCode=0
Nov 02 22:30:00.894488 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.894415    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"
Nov 02 22:30:00.894488 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.894452    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" event={"ID":"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612","Type":"ContainerDied","Data":"a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe"}
Nov 02 22:30:00.894574 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.894488    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt" event={"ID":"57c65c87-4bcb-4a5c-b4d1-396f2a7f3612","Type":"ContainerDied","Data":"eae3be9ff06920515ad56fc7e1c1f55dde01964a4979c29e63fb883a0fc8b572"}
Nov 02 22:30:00.894574 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.894511    4528 scope.go:122] "RemoveContainer" containerID="a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe"
Nov 02 22:30:00.912600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.910291    4528 scope.go:122] "RemoveContainer" containerID="a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe"
Nov 02 22:30:00.912600 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:00.910599    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe\": container with ID starting with a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe not found: ID does not exist" containerID="a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe"
Nov 02 22:30:00.912600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.910632    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe"} err="failed to get container status \"a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe\": rpc error: code = NotFound desc = could not find container \"a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe\": container with ID starting with a45ec21792771b04d6cb3f1dc5d9f21db4df2893ed5325d6a0041ccff9e91dfe not found: ID does not exist"
Nov 02 22:30:00.917920 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:30:00.917894    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd921a58d_7bdc_4793_bf83_d53ffb571b0f.slice/crio-bd2ba81a16b421acd31804e05efa69d42dd009eaed1cc73a70b1a03b22dbb6c7 WatchSource:0}: Error finding container bd2ba81a16b421acd31804e05efa69d42dd009eaed1cc73a70b1a03b22dbb6c7: Status 404 returned error can't find the container with id bd2ba81a16b421acd31804e05efa69d42dd009eaed1cc73a70b1a03b22dbb6c7
Nov 02 22:30:00.920476 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:30:00.920439    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991fa00a_364a_4544_a478_66e3eac54f46.slice/crio-2d972d7c49a8be3cc595d27a221d4d988cbc78e7c07267cc10f3f38a79e7875d WatchSource:0}: Error finding container 2d972d7c49a8be3cc595d27a221d4d988cbc78e7c07267cc10f3f38a79e7875d: Status 404 returned error can't find the container with id 2d972d7c49a8be3cc595d27a221d4d988cbc78e7c07267cc10f3f38a79e7875d
Nov 02 22:30:00.924298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.924215    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2bee1ea-ab9d-421f-badf-59117f3de3ac" path="/var/lib/kubelet/pods/d2bee1ea-ab9d-421f-badf-59117f3de3ac/volumes"
Nov 02 22:30:00.924606 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.924580    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"]
Nov 02 22:30:00.924694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.924634    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"]
Nov 02 22:30:00.933770 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.933680    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"]
Nov 02 22:30:00.940352 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:30:00.940326    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa1bf8f_c68e_421e_9b92_b5c49a6c96da.slice/crio-d1d3c2572257bc503b6f3b7f3b830c0e9e0da2879009c5cda67e54b24927b7fe WatchSource:0}: Error finding container d1d3c2572257bc503b6f3b7f3b830c0e9e0da2879009c5cda67e54b24927b7fe: Status 404 returned error can't find the container with id d1d3c2572257bc503b6f3b7f3b830c0e9e0da2879009c5cda67e54b24927b7fe
Nov 02 22:30:00.966286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:00.966258    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"]
Nov 02 22:30:00.974498 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:30:00.974471    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc251060_916a_4354_9ad9_35a9fd77fada.slice/crio-c6c312c62a399b18b6a9eb2575e2de9d29da1d9ae748d4f2b091f3a07821ee19 WatchSource:0}: Error finding container c6c312c62a399b18b6a9eb2575e2de9d29da1d9ae748d4f2b091f3a07821ee19: Status 404 returned error can't find the container with id c6c312c62a399b18b6a9eb2575e2de9d29da1d9ae748d4f2b091f3a07821ee19
Nov 02 22:30:01.059279 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.059255    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"]
Nov 02 22:30:01.062022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.061986    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/premium-w4-d6cb38-kserve-57466db89d-fssbt"]
Nov 02 22:30:01.114493 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.114472    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:30:01.248141 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.248059    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-xks-ca\") pod \"22a24acc-dc6f-4ef9-b75c-332992234503\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") "
Nov 02 22:30:01.248489 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.248220    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-openshift-ca\") pod \"22a24acc-dc6f-4ef9-b75c-332992234503\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") "
Nov 02 22:30:01.248489 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.248311    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a24acc-dc6f-4ef9-b75c-332992234503-kube-api-access-shf7b\" (UniqueName: \"kubernetes.io/projected/22a24acc-dc6f-4ef9-b75c-332992234503-kube-api-access-shf7b\") pod \"22a24acc-dc6f-4ef9-b75c-332992234503\" (UID: \"22a24acc-dc6f-4ef9-b75c-332992234503\") "
Nov 02 22:30:01.248787 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.248762    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-openshift-ca" pod "22a24acc-dc6f-4ef9-b75c-332992234503" (UID: "22a24acc-dc6f-4ef9-b75c-332992234503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:30:01.255435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.255410    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-xks-ca" pod "22a24acc-dc6f-4ef9-b75c-332992234503" (UID: "22a24acc-dc6f-4ef9-b75c-332992234503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:30:01.255550 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.255433    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a24acc-dc6f-4ef9-b75c-332992234503-kube-api-access-shf7b" pod "22a24acc-dc6f-4ef9-b75c-332992234503" (UID: "22a24acc-dc6f-4ef9-b75c-332992234503"). InnerVolumeSpecName "kube-api-access-shf7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:30:01.349825 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.349798    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:01.349825 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.349822    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a24acc-dc6f-4ef9-b75c-332992234503-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:01.349976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.349833    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shf7b\" (UniqueName: \"kubernetes.io/projected/22a24acc-dc6f-4ef9-b75c-332992234503-kube-api-access-shf7b\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:01.916050 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.916011    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" event={"ID":"991fa00a-364a-4544-a478-66e3eac54f46","Type":"ContainerStarted","Data":"cabb78e13c8bd6fffe952d9e9294685cf053683c9297ba17013253f0896d0480"}
Nov 02 22:30:01.916391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.916083    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" event={"ID":"991fa00a-364a-4544-a478-66e3eac54f46","Type":"ContainerStarted","Data":"2d972d7c49a8be3cc595d27a221d4d988cbc78e7c07267cc10f3f38a79e7875d"}
Nov 02 22:30:01.919537 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.919095    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" event={"ID":"cc251060-916a-4354-9ad9-35a9fd77fada","Type":"ContainerStarted","Data":"f23f3401a6bc5d8e09f2664209bfd3324820bccf2a88efa263880d8a4501df32"}
Nov 02 22:30:01.919537 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.919233    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" event={"ID":"cc251060-916a-4354-9ad9-35a9fd77fada","Type":"ContainerStarted","Data":"c6c312c62a399b18b6a9eb2575e2de9d29da1d9ae748d4f2b091f3a07821ee19"}
Nov 02 22:30:01.920608 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.920585    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"
Nov 02 22:30:01.920705 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.920605    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml" event={"ID":"22a24acc-dc6f-4ef9-b75c-332992234503","Type":"ContainerDied","Data":"0b0945c39b2ac378fb0a6e4abaff6aada655464445deee935011f3b9d2d883eb"}
Nov 02 22:30:01.920705 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.920650    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b0945c39b2ac378fb0a6e4abaff6aada655464445deee935011f3b9d2d883eb"
Nov 02 22:30:01.922160 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.922134    4528 generic.go:356] "Generic (PLEG): container finished" podID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerID="ba54045d3ddab7138aaa3a149753dd4b4246c21535055815826156541b72d8a7" exitCode=60
Nov 02 22:30:01.922272 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.922214    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" event={"ID":"a8cff92e-d2fa-4e9d-8275-1d9e401af54c","Type":"ContainerDied","Data":"ba54045d3ddab7138aaa3a149753dd4b4246c21535055815826156541b72d8a7"}
Nov 02 22:30:01.922410 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.922394    4528 scope.go:122] "RemoveContainer" containerID="ba54045d3ddab7138aaa3a149753dd4b4246c21535055815826156541b72d8a7"
Nov 02 22:30:01.923919 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.923902    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v" event={"ID":"d921a58d-7bdc-4793-bf83-d53ffb571b0f","Type":"ContainerStarted","Data":"4514ca2403f75a9e09c3ef1f742d09a2553bf36d13d2eeb71ed8bcec742ba13e"}
Nov 02 22:30:01.924011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.923942    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v" event={"ID":"d921a58d-7bdc-4793-bf83-d53ffb571b0f","Type":"ContainerStarted","Data":"bd2ba81a16b421acd31804e05efa69d42dd009eaed1cc73a70b1a03b22dbb6c7"}
Nov 02 22:30:01.925642 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.925618    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" event={"ID":"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da","Type":"ContainerStarted","Data":"57cea8d60df0ee23175249e1d62e286b69f98e50d1d2289d09bd0e087f28eedd"}
Nov 02 22:30:01.925721 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.925668    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" event={"ID":"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da","Type":"ContainerStarted","Data":"d1d3c2572257bc503b6f3b7f3b830c0e9e0da2879009c5cda67e54b24927b7fe"}
Nov 02 22:30:01.925822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:01.925804    4528 scope.go:122] "RemoveContainer" containerID="57cea8d60df0ee23175249e1d62e286b69f98e50d1d2289d09bd0e087f28eedd"
Nov 02 22:30:02.031214 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.031164    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" podStartSLOduration=2.031150601 podStartE2EDuration="2.031150601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:02.028762905 +0000 UTC m=+4889.292299222" watchObservedRunningTime="2026-11-02 22:30:02.031150601 +0000 UTC m=+4889.294686916"
Nov 02 22:30:02.141898 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.141853    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" podStartSLOduration=2.141840035 podStartE2EDuration="2.141840035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:02.140408731 +0000 UTC m=+4889.403945191" watchObservedRunningTime="2026-11-02 22:30:02.141840035 +0000 UTC m=+4889.405376350"
Nov 02 22:30:02.479312 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.479263    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v" podStartSLOduration=2.47924975 podStartE2EDuration="2.47924975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:02.47666907 +0000 UTC m=+4889.740205387" watchObservedRunningTime="2026-11-02 22:30:02.47924975 +0000 UTC m=+4889.742786066"
Nov 02 22:30:02.913198 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.913165    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" path="/var/lib/kubelet/pods/57c65c87-4bcb-4a5c-b4d1-396f2a7f3612/volumes"
Nov 02 22:30:02.937625 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.937589    4528 generic.go:356] "Generic (PLEG): container finished" podID="991fa00a-364a-4544-a478-66e3eac54f46" containerID="cabb78e13c8bd6fffe952d9e9294685cf053683c9297ba17013253f0896d0480" exitCode=60
Nov 02 22:30:02.937776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.937674    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" event={"ID":"991fa00a-364a-4544-a478-66e3eac54f46","Type":"ContainerDied","Data":"cabb78e13c8bd6fffe952d9e9294685cf053683c9297ba17013253f0896d0480"}
Nov 02 22:30:02.937894 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.937875    4528 scope.go:122] "RemoveContainer" containerID="cabb78e13c8bd6fffe952d9e9294685cf053683c9297ba17013253f0896d0480"
Nov 02 22:30:02.939326 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.939205    4528 generic.go:356] "Generic (PLEG): container finished" podID="cc251060-916a-4354-9ad9-35a9fd77fada" containerID="f23f3401a6bc5d8e09f2664209bfd3324820bccf2a88efa263880d8a4501df32" exitCode=60
Nov 02 22:30:02.939326 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.939287    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" event={"ID":"cc251060-916a-4354-9ad9-35a9fd77fada","Type":"ContainerDied","Data":"f23f3401a6bc5d8e09f2664209bfd3324820bccf2a88efa263880d8a4501df32"}
Nov 02 22:30:02.939448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.939434    4528 scope.go:122] "RemoveContainer" containerID="f23f3401a6bc5d8e09f2664209bfd3324820bccf2a88efa263880d8a4501df32"
Nov 02 22:30:02.942155 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.941796    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" event={"ID":"a8cff92e-d2fa-4e9d-8275-1d9e401af54c","Type":"ContainerStarted","Data":"d63a7cb36c807a6d14d52c28f8207716d3a0dd0c9953254154dc3cb23c145467"}
Nov 02 22:30:02.943286 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.943259    4528 generic.go:356] "Generic (PLEG): container finished" podID="d921a58d-7bdc-4793-bf83-d53ffb571b0f" containerID="4514ca2403f75a9e09c3ef1f742d09a2553bf36d13d2eeb71ed8bcec742ba13e" exitCode=60
Nov 02 22:30:02.943399 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.943304    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v" event={"ID":"d921a58d-7bdc-4793-bf83-d53ffb571b0f","Type":"ContainerDied","Data":"4514ca2403f75a9e09c3ef1f742d09a2553bf36d13d2eeb71ed8bcec742ba13e"}
Nov 02 22:30:02.943536 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.943520    4528 scope.go:122] "RemoveContainer" containerID="4514ca2403f75a9e09c3ef1f742d09a2553bf36d13d2eeb71ed8bcec742ba13e"
Nov 02 22:30:02.945103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.945084    4528 generic.go:356] "Generic (PLEG): container finished" podID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerID="57cea8d60df0ee23175249e1d62e286b69f98e50d1d2289d09bd0e087f28eedd" exitCode=60
Nov 02 22:30:02.945222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.945140    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" event={"ID":"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da","Type":"ContainerDied","Data":"57cea8d60df0ee23175249e1d62e286b69f98e50d1d2289d09bd0e087f28eedd"}
Nov 02 22:30:02.945222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:02.945190    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" event={"ID":"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da","Type":"ContainerStarted","Data":"1583875aae26536e645bd47d9628590dae3e1f3a00544c94ee248c1bfec3f513"}
Nov 02 22:30:03.163275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.163104    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" podStartSLOduration=3.163088086 podStartE2EDuration="3.163088086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:03.162266815 +0000 UTC m=+4890.425803132" watchObservedRunningTime="2026-11-02 22:30:03.163088086 +0000 UTC m=+4890.426624404"
Nov 02 22:30:03.276476 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.276429    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" podStartSLOduration=3.276414799 podStartE2EDuration="3.276414799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:03.274771638 +0000 UTC m=+4890.538307954" watchObservedRunningTime="2026-11-02 22:30:03.276414799 +0000 UTC m=+4890.539951137"
Nov 02 22:30:03.954038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.954009    4528 generic.go:356] "Generic (PLEG): container finished" podID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerID="1583875aae26536e645bd47d9628590dae3e1f3a00544c94ee248c1bfec3f513" exitCode=60
Nov 02 22:30:03.954448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.954092    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" event={"ID":"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da","Type":"ContainerDied","Data":"1583875aae26536e645bd47d9628590dae3e1f3a00544c94ee248c1bfec3f513"}
Nov 02 22:30:03.954448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.954174    4528 scope.go:122] "RemoveContainer" containerID="57cea8d60df0ee23175249e1d62e286b69f98e50d1d2289d09bd0e087f28eedd"
Nov 02 22:30:03.954448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.954378    4528 scope.go:122] "RemoveContainer" containerID="1583875aae26536e645bd47d9628590dae3e1f3a00544c94ee248c1bfec3f513"
Nov 02 22:30:03.954674 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:03.954650    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp_odh-ai-gateway-infra(aaa1bf8f-c68e-421e-9b92-b5c49a6c96da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da"
Nov 02 22:30:03.956060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.956037    4528 generic.go:356] "Generic (PLEG): container finished" podID="991fa00a-364a-4544-a478-66e3eac54f46" containerID="d519abf967e4abbd4008683cc8a7a80084b6fa41de1b6f4f5a7b8ded046f4df3" exitCode=60
Nov 02 22:30:03.956177 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.956123    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" event={"ID":"991fa00a-364a-4544-a478-66e3eac54f46","Type":"ContainerDied","Data":"d519abf967e4abbd4008683cc8a7a80084b6fa41de1b6f4f5a7b8ded046f4df3"}
Nov 02 22:30:03.956366 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.956353    4528 scope.go:122] "RemoveContainer" containerID="d519abf967e4abbd4008683cc8a7a80084b6fa41de1b6f4f5a7b8ded046f4df3"
Nov 02 22:30:03.956634 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:03.956613    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5_odh-ai-gateway-infra(991fa00a-364a-4544-a478-66e3eac54f46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" podUID="991fa00a-364a-4544-a478-66e3eac54f46"
Nov 02 22:30:03.958037 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.958017    4528 generic.go:356] "Generic (PLEG): container finished" podID="cc251060-916a-4354-9ad9-35a9fd77fada" containerID="987fec8ae0ee619018a3978c32c325fb97dbe8152d222b58d3b262eb23dade1d" exitCode=60
Nov 02 22:30:03.958142 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.958104    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" event={"ID":"cc251060-916a-4354-9ad9-35a9fd77fada","Type":"ContainerDied","Data":"987fec8ae0ee619018a3978c32c325fb97dbe8152d222b58d3b262eb23dade1d"}
Nov 02 22:30:03.958349 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.958311    4528 scope.go:122] "RemoveContainer" containerID="987fec8ae0ee619018a3978c32c325fb97dbe8152d222b58d3b262eb23dade1d"
Nov 02 22:30:03.958564 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:03.958538    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv_odh-ai-gateway-infra(cc251060-916a-4354-9ad9-35a9fd77fada)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" podUID="cc251060-916a-4354-9ad9-35a9fd77fada"
Nov 02 22:30:03.960044 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.960027    4528 generic.go:356] "Generic (PLEG): container finished" podID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerID="d63a7cb36c807a6d14d52c28f8207716d3a0dd0c9953254154dc3cb23c145467" exitCode=60
Nov 02 22:30:03.960150 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.960127    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" event={"ID":"a8cff92e-d2fa-4e9d-8275-1d9e401af54c","Type":"ContainerDied","Data":"d63a7cb36c807a6d14d52c28f8207716d3a0dd0c9953254154dc3cb23c145467"}
Nov 02 22:30:03.960318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.960302    4528 scope.go:122] "RemoveContainer" containerID="d63a7cb36c807a6d14d52c28f8207716d3a0dd0c9953254154dc3cb23c145467"
Nov 02 22:30:03.960534 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:03.960514    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl_odh-ai-gateway-infra(a8cff92e-d2fa-4e9d-8275-1d9e401af54c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c"
Nov 02 22:30:03.961935 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.961917    4528 generic.go:356] "Generic (PLEG): container finished" podID="d921a58d-7bdc-4793-bf83-d53ffb571b0f" containerID="95a0fd65cad04892f6fc1ea2edf6afdf6c6904bef60d7d8538dd05786041a6d6" exitCode=60
Nov 02 22:30:03.962027 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.961955    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v" event={"ID":"d921a58d-7bdc-4793-bf83-d53ffb571b0f","Type":"ContainerDied","Data":"95a0fd65cad04892f6fc1ea2edf6afdf6c6904bef60d7d8538dd05786041a6d6"}
Nov 02 22:30:03.962176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:03.962163    4528 scope.go:122] "RemoveContainer" containerID="95a0fd65cad04892f6fc1ea2edf6afdf6c6904bef60d7d8538dd05786041a6d6"
Nov 02 22:30:03.962387 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:03.962371    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v_odh-ai-gateway-infra(d921a58d-7bdc-4793-bf83-d53ffb571b0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v" podUID="d921a58d-7bdc-4793-bf83-d53ffb571b0f"
Nov 02 22:30:04.010101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:04.010072    4528 scope.go:122] "RemoveContainer" containerID="cabb78e13c8bd6fffe952d9e9294685cf053683c9297ba17013253f0896d0480"
Nov 02 22:30:04.026087 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:04.026066    4528 scope.go:122] "RemoveContainer" containerID="f23f3401a6bc5d8e09f2664209bfd3324820bccf2a88efa263880d8a4501df32"
Nov 02 22:30:04.039030 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:04.039010    4528 scope.go:122] "RemoveContainer" containerID="ba54045d3ddab7138aaa3a149753dd4b4246c21535055815826156541b72d8a7"
Nov 02 22:30:04.050582 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:04.050558    4528 scope.go:122] "RemoveContainer" containerID="4514ca2403f75a9e09c3ef1f742d09a2553bf36d13d2eeb71ed8bcec742ba13e"
Nov 02 22:30:04.971739 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:04.971711    4528 scope.go:122] "RemoveContainer" containerID="d63a7cb36c807a6d14d52c28f8207716d3a0dd0c9953254154dc3cb23c145467"
Nov 02 22:30:04.972185 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:04.972002    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl_odh-ai-gateway-infra(a8cff92e-d2fa-4e9d-8275-1d9e401af54c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c"
Nov 02 22:30:04.973309 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:04.973292    4528 scope.go:122] "RemoveContainer" containerID="95a0fd65cad04892f6fc1ea2edf6afdf6c6904bef60d7d8538dd05786041a6d6"
Nov 02 22:30:04.973553 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:04.973533    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v_odh-ai-gateway-infra(d921a58d-7bdc-4793-bf83-d53ffb571b0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v" podUID="d921a58d-7bdc-4793-bf83-d53ffb571b0f"
Nov 02 22:30:04.974735 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:04.974710    4528 scope.go:122] "RemoveContainer" containerID="1583875aae26536e645bd47d9628590dae3e1f3a00544c94ee248c1bfec3f513"
Nov 02 22:30:04.974965 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:04.974944    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp_odh-ai-gateway-infra(aaa1bf8f-c68e-421e-9b92-b5c49a6c96da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da"
Nov 02 22:30:04.976247 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:04.976229    4528 scope.go:122] "RemoveContainer" containerID="d519abf967e4abbd4008683cc8a7a80084b6fa41de1b6f4f5a7b8ded046f4df3"
Nov 02 22:30:04.976434 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:04.976413    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5_odh-ai-gateway-infra(991fa00a-364a-4544-a478-66e3eac54f46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" podUID="991fa00a-364a-4544-a478-66e3eac54f46"
Nov 02 22:30:04.977710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:04.977692    4528 scope.go:122] "RemoveContainer" containerID="987fec8ae0ee619018a3978c32c325fb97dbe8152d222b58d3b262eb23dade1d"
Nov 02 22:30:04.977876 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:04.977859    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv_odh-ai-gateway-infra(cc251060-916a-4354-9ad9-35a9fd77fada)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" podUID="cc251060-916a-4354-9ad9-35a9fd77fada"
Nov 02 22:30:13.535940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:13.535903    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"]
Nov 02 22:30:13.691005 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:13.690982    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:13.776617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:13.776587    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d921a58d-7bdc-4793-bf83-d53ffb571b0f-kube-api-access-2djfl\" (UniqueName: \"kubernetes.io/projected/d921a58d-7bdc-4793-bf83-d53ffb571b0f-kube-api-access-2djfl\") pod \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\" (UID: \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\") "
Nov 02 22:30:13.776784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:13.776657    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d921a58d-7bdc-4793-bf83-d53ffb571b0f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921a58d-7bdc-4793-bf83-d53ffb571b0f-maas-api-service-ca\") pod \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\" (UID: \"d921a58d-7bdc-4793-bf83-d53ffb571b0f\") "
Nov 02 22:30:13.777061 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:13.777034    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d921a58d-7bdc-4793-bf83-d53ffb571b0f-maas-api-service-ca" pod "d921a58d-7bdc-4793-bf83-d53ffb571b0f" (UID: "d921a58d-7bdc-4793-bf83-d53ffb571b0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:30:13.778603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:13.778584    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d921a58d-7bdc-4793-bf83-d53ffb571b0f-kube-api-access-2djfl" pod "d921a58d-7bdc-4793-bf83-d53ffb571b0f" (UID: "d921a58d-7bdc-4793-bf83-d53ffb571b0f"). InnerVolumeSpecName "kube-api-access-2djfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:30:13.878571 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:13.878486    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2djfl\" (UniqueName: \"kubernetes.io/projected/d921a58d-7bdc-4793-bf83-d53ffb571b0f-kube-api-access-2djfl\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:13.878571 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:13.878519    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d921a58d-7bdc-4793-bf83-d53ffb571b0f-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:13.906667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:13.906634    4528 scope.go:122] "RemoveContainer" containerID="1583875aae26536e645bd47d9628590dae3e1f3a00544c94ee248c1bfec3f513"
Nov 02 22:30:14.033145 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:14.033099    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"
Nov 02 22:30:14.033145 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:14.033134    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v" event={"ID":"d921a58d-7bdc-4793-bf83-d53ffb571b0f","Type":"ContainerDied","Data":"bd2ba81a16b421acd31804e05efa69d42dd009eaed1cc73a70b1a03b22dbb6c7"}
Nov 02 22:30:14.033345 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:14.033197    4528 scope.go:122] "RemoveContainer" containerID="95a0fd65cad04892f6fc1ea2edf6afdf6c6904bef60d7d8538dd05786041a6d6"
Nov 02 22:30:14.113241 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:14.113207    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:30:14.199013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:14.198925    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"]
Nov 02 22:30:14.201645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:14.201625    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894310-t664v"]
Nov 02 22:30:14.906402 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:14.906372    4528 scope.go:122] "RemoveContainer" containerID="d63a7cb36c807a6d14d52c28f8207716d3a0dd0c9953254154dc3cb23c145467"
Nov 02 22:30:14.912589 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:14.912559    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d921a58d-7bdc-4793-bf83-d53ffb571b0f" path="/var/lib/kubelet/pods/d921a58d-7bdc-4793-bf83-d53ffb571b0f/volumes"
Nov 02 22:30:15.041857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:15.041829    4528 generic.go:356] "Generic (PLEG): container finished" podID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerID="8bd82602a5726c94c89affe3310e235363ade849ba33ef2f3275d3323e2fa473" exitCode=60
Nov 02 22:30:15.042022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:15.041882    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" event={"ID":"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da","Type":"ContainerDied","Data":"8bd82602a5726c94c89affe3310e235363ade849ba33ef2f3275d3323e2fa473"}
Nov 02 22:30:15.042022 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:15.041970    4528 scope.go:122] "RemoveContainer" containerID="1583875aae26536e645bd47d9628590dae3e1f3a00544c94ee248c1bfec3f513"
Nov 02 22:30:15.042384 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:15.042361    4528 scope.go:122] "RemoveContainer" containerID="8bd82602a5726c94c89affe3310e235363ade849ba33ef2f3275d3323e2fa473"
Nov 02 22:30:15.042718 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:15.042685    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp_odh-ai-gateway-infra(aaa1bf8f-c68e-421e-9b92-b5c49a6c96da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da"
Nov 02 22:30:16.049391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.049356    4528 generic.go:356] "Generic (PLEG): container finished" podID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerID="dcfdf7f1ea46e442a06a3e93b2a874424df85a17671572430dde112fde178b30" exitCode=60
Nov 02 22:30:16.049844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.049426    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" event={"ID":"a8cff92e-d2fa-4e9d-8275-1d9e401af54c","Type":"ContainerDied","Data":"dcfdf7f1ea46e442a06a3e93b2a874424df85a17671572430dde112fde178b30"}
Nov 02 22:30:16.049844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.049480    4528 scope.go:122] "RemoveContainer" containerID="d63a7cb36c807a6d14d52c28f8207716d3a0dd0c9953254154dc3cb23c145467"
Nov 02 22:30:16.049844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.049816    4528 scope.go:122] "RemoveContainer" containerID="dcfdf7f1ea46e442a06a3e93b2a874424df85a17671572430dde112fde178b30"
Nov 02 22:30:16.050150 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:16.050101    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl_odh-ai-gateway-infra(a8cff92e-d2fa-4e9d-8275-1d9e401af54c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c"
Nov 02 22:30:16.168621 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.168591    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"]
Nov 02 22:30:16.310310 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.310247    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:16.413183 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.413152    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-kube-api-access-xck7p\" (UniqueName: \"kubernetes.io/projected/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-kube-api-access-xck7p\") pod \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\" (UID: \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\") "
Nov 02 22:30:16.413388 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.413298    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-maas-api-service-ca\") pod \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\" (UID: \"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da\") "
Nov 02 22:30:16.413652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.413628    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-maas-api-service-ca" pod "aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" (UID: "aaa1bf8f-c68e-421e-9b92-b5c49a6c96da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:30:16.415297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.415276    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-kube-api-access-xck7p" pod "aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" (UID: "aaa1bf8f-c68e-421e-9b92-b5c49a6c96da"). InnerVolumeSpecName "kube-api-access-xck7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:30:16.514726 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.514696    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xck7p\" (UniqueName: \"kubernetes.io/projected/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-kube-api-access-xck7p\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:16.514726 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:16.514725    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:17.062464 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.062435    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"
Nov 02 22:30:17.062913 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.062466    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp" event={"ID":"aaa1bf8f-c68e-421e-9b92-b5c49a6c96da","Type":"ContainerDied","Data":"d1d3c2572257bc503b6f3b7f3b830c0e9e0da2879009c5cda67e54b24927b7fe"}
Nov 02 22:30:17.062913 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.062523    4528 scope.go:122] "RemoveContainer" containerID="8bd82602a5726c94c89affe3310e235363ade849ba33ef2f3275d3323e2fa473"
Nov 02 22:30:17.173099 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.173020    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"]
Nov 02 22:30:17.226723 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.226691    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"]
Nov 02 22:30:17.232784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.232748    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6e535d-29894310-5vjmp"]
Nov 02 22:30:17.314386 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.314364    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:17.426256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.426174    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-kube-api-access-dr4cm\" (UniqueName: \"kubernetes.io/projected/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-kube-api-access-dr4cm\") pod \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\" (UID: \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\") "
Nov 02 22:30:17.426420 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.426317    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-maas-api-service-ca\") pod \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\" (UID: \"a8cff92e-d2fa-4e9d-8275-1d9e401af54c\") "
Nov 02 22:30:17.426684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.426661    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-maas-api-service-ca" pod "a8cff92e-d2fa-4e9d-8275-1d9e401af54c" (UID: "a8cff92e-d2fa-4e9d-8275-1d9e401af54c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:30:17.428332 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.428311    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-kube-api-access-dr4cm" pod "a8cff92e-d2fa-4e9d-8275-1d9e401af54c" (UID: "a8cff92e-d2fa-4e9d-8275-1d9e401af54c"). InnerVolumeSpecName "kube-api-access-dr4cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:30:17.528006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.527955    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:17.528006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.527999    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr4cm\" (UniqueName: \"kubernetes.io/projected/a8cff92e-d2fa-4e9d-8275-1d9e401af54c-kube-api-access-dr4cm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:17.906415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:17.906378    4528 scope.go:122] "RemoveContainer" containerID="d519abf967e4abbd4008683cc8a7a80084b6fa41de1b6f4f5a7b8ded046f4df3"
Nov 02 22:30:18.070838 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:18.070813    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"
Nov 02 22:30:18.071288 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:18.070853    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl" event={"ID":"a8cff92e-d2fa-4e9d-8275-1d9e401af54c","Type":"ContainerDied","Data":"6ef6d9d3678be6a1e4eeb659e65e46926579335dbbfc3c49f7623a3e7b059b02"}
Nov 02 22:30:18.071288 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:18.070919    4528 scope.go:122] "RemoveContainer" containerID="dcfdf7f1ea46e442a06a3e93b2a874424df85a17671572430dde112fde178b30"
Nov 02 22:30:18.238527 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:18.238503    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"]
Nov 02 22:30:18.288516 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:18.288492    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894310-4bcxl"]
Nov 02 22:30:18.906572 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:18.906540    4528 scope.go:122] "RemoveContainer" containerID="987fec8ae0ee619018a3978c32c325fb97dbe8152d222b58d3b262eb23dade1d"
Nov 02 22:30:18.913311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:18.913284    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" path="/var/lib/kubelet/pods/a8cff92e-d2fa-4e9d-8275-1d9e401af54c/volumes"
Nov 02 22:30:18.913771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:18.913754    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" path="/var/lib/kubelet/pods/aaa1bf8f-c68e-421e-9b92-b5c49a6c96da/volumes"
Nov 02 22:30:19.080595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:19.080562    4528 generic.go:356] "Generic (PLEG): container finished" podID="991fa00a-364a-4544-a478-66e3eac54f46" containerID="30af84a9bfa16ad997364015d3a9b3943988d79b0c2fa2c79b8872a977bc3804" exitCode=60
Nov 02 22:30:19.081060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:19.080637    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" event={"ID":"991fa00a-364a-4544-a478-66e3eac54f46","Type":"ContainerDied","Data":"30af84a9bfa16ad997364015d3a9b3943988d79b0c2fa2c79b8872a977bc3804"}
Nov 02 22:30:19.081060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:19.080708    4528 scope.go:122] "RemoveContainer" containerID="d519abf967e4abbd4008683cc8a7a80084b6fa41de1b6f4f5a7b8ded046f4df3"
Nov 02 22:30:19.081060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:19.081049    4528 scope.go:122] "RemoveContainer" containerID="30af84a9bfa16ad997364015d3a9b3943988d79b0c2fa2c79b8872a977bc3804"
Nov 02 22:30:19.081364 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:19.081338    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5_odh-ai-gateway-infra(991fa00a-364a-4544-a478-66e3eac54f46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" podUID="991fa00a-364a-4544-a478-66e3eac54f46"
Nov 02 22:30:20.093914 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.093878    4528 generic.go:356] "Generic (PLEG): container finished" podID="cc251060-916a-4354-9ad9-35a9fd77fada" containerID="80609c5a0787135674f3b8001d957c77d8814a2d456cd02521869290ac8cefa7" exitCode=60
Nov 02 22:30:20.094696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.094668    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" event={"ID":"cc251060-916a-4354-9ad9-35a9fd77fada","Type":"ContainerDied","Data":"80609c5a0787135674f3b8001d957c77d8814a2d456cd02521869290ac8cefa7"}
Nov 02 22:30:20.094881 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.094864    4528 scope.go:122] "RemoveContainer" containerID="987fec8ae0ee619018a3978c32c325fb97dbe8152d222b58d3b262eb23dade1d"
Nov 02 22:30:20.095459 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.095433    4528 scope.go:122] "RemoveContainer" containerID="80609c5a0787135674f3b8001d957c77d8814a2d456cd02521869290ac8cefa7"
Nov 02 22:30:20.095780 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:20.095755    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv_odh-ai-gateway-infra(cc251060-916a-4354-9ad9-35a9fd77fada)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" podUID="cc251060-916a-4354-9ad9-35a9fd77fada"
Nov 02 22:30:20.229675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.229643    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"]
Nov 02 22:30:20.375502 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.375480    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:20.452749 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.452716    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991fa00a-364a-4544-a478-66e3eac54f46-kube-api-access-v7mnr\" (UniqueName: \"kubernetes.io/projected/991fa00a-364a-4544-a478-66e3eac54f46-kube-api-access-v7mnr\") pod \"991fa00a-364a-4544-a478-66e3eac54f46\" (UID: \"991fa00a-364a-4544-a478-66e3eac54f46\") "
Nov 02 22:30:20.452920 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.452827    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991fa00a-364a-4544-a478-66e3eac54f46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991fa00a-364a-4544-a478-66e3eac54f46-maas-api-service-ca\") pod \"991fa00a-364a-4544-a478-66e3eac54f46\" (UID: \"991fa00a-364a-4544-a478-66e3eac54f46\") "
Nov 02 22:30:20.453239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.453209    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991fa00a-364a-4544-a478-66e3eac54f46-maas-api-service-ca" pod "991fa00a-364a-4544-a478-66e3eac54f46" (UID: "991fa00a-364a-4544-a478-66e3eac54f46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:30:20.454887 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.454860    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991fa00a-364a-4544-a478-66e3eac54f46-kube-api-access-v7mnr" pod "991fa00a-364a-4544-a478-66e3eac54f46" (UID: "991fa00a-364a-4544-a478-66e3eac54f46"). InnerVolumeSpecName "kube-api-access-v7mnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:30:20.553720 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.553688    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mnr\" (UniqueName: \"kubernetes.io/projected/991fa00a-364a-4544-a478-66e3eac54f46-kube-api-access-v7mnr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:20.553720 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:20.553717    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991fa00a-364a-4544-a478-66e3eac54f46-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:21.105847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.105812    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"
Nov 02 22:30:21.106305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.105815    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5" event={"ID":"991fa00a-364a-4544-a478-66e3eac54f46","Type":"ContainerDied","Data":"2d972d7c49a8be3cc595d27a221d4d988cbc78e7c07267cc10f3f38a79e7875d"}
Nov 02 22:30:21.106305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.105972    4528 scope.go:122] "RemoveContainer" containerID="30af84a9bfa16ad997364015d3a9b3943988d79b0c2fa2c79b8872a977bc3804"
Nov 02 22:30:21.239745 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.239667    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"]
Nov 02 22:30:21.272784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.272749    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"]
Nov 02 22:30:21.288503 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.288466    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a1d751-29894310-nt4h5"]
Nov 02 22:30:21.382466 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.382444    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:21.562544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.562514    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc251060-916a-4354-9ad9-35a9fd77fada-kube-api-access-n9vp2\" (UniqueName: \"kubernetes.io/projected/cc251060-916a-4354-9ad9-35a9fd77fada-kube-api-access-n9vp2\") pod \"cc251060-916a-4354-9ad9-35a9fd77fada\" (UID: \"cc251060-916a-4354-9ad9-35a9fd77fada\") "
Nov 02 22:30:21.562722 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.562561    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc251060-916a-4354-9ad9-35a9fd77fada-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc251060-916a-4354-9ad9-35a9fd77fada-maas-api-service-ca\") pod \"cc251060-916a-4354-9ad9-35a9fd77fada\" (UID: \"cc251060-916a-4354-9ad9-35a9fd77fada\") "
Nov 02 22:30:21.562938 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.562916    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc251060-916a-4354-9ad9-35a9fd77fada-maas-api-service-ca" pod "cc251060-916a-4354-9ad9-35a9fd77fada" (UID: "cc251060-916a-4354-9ad9-35a9fd77fada"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:30:21.564657 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.564629    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc251060-916a-4354-9ad9-35a9fd77fada-kube-api-access-n9vp2" pod "cc251060-916a-4354-9ad9-35a9fd77fada" (UID: "cc251060-916a-4354-9ad9-35a9fd77fada"). InnerVolumeSpecName "kube-api-access-n9vp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:30:21.639907 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.639873    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"]
Nov 02 22:30:21.663385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.663360    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9vp2\" (UniqueName: \"kubernetes.io/projected/cc251060-916a-4354-9ad9-35a9fd77fada-kube-api-access-n9vp2\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:21.663385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:21.663385    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc251060-916a-4354-9ad9-35a9fd77fada-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:22.115620 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:22.115586    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"
Nov 02 22:30:22.116131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:22.115591    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv" event={"ID":"cc251060-916a-4354-9ad9-35a9fd77fada","Type":"ContainerDied","Data":"c6c312c62a399b18b6a9eb2575e2de9d29da1d9ae748d4f2b091f3a07821ee19"}
Nov 02 22:30:22.116131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:22.115728    4528 scope.go:122] "RemoveContainer" containerID="80609c5a0787135674f3b8001d957c77d8814a2d456cd02521869290ac8cefa7"
Nov 02 22:30:22.279823 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:22.279788    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"]
Nov 02 22:30:22.285071 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:22.285046    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894310-g8szv"]
Nov 02 22:30:22.912584 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:22.912549    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991fa00a-364a-4544-a478-66e3eac54f46" path="/var/lib/kubelet/pods/991fa00a-364a-4544-a478-66e3eac54f46/volumes"
Nov 02 22:30:22.912963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:22.912949    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" path="/var/lib/kubelet/pods/cc251060-916a-4354-9ad9-35a9fd77fada/volumes"
Nov 02 22:30:26.640715 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:26.640667    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t" podUID="cfa258a7-12ac-4547-80ce-3e4826883324" containerName="payload-processing" containerID="cri-o://63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837" gracePeriod=25
Nov 02 22:30:30.543747 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:30.543708    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t" podUID="cfa258a7-12ac-4547-80ce-3e4826883324" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:30:33.095665 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:33.095623    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:30:38.046764 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:38.046731    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"]
Nov 02 22:30:38.096775 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:38.096741    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58996c83-lh5bc"]
Nov 02 22:30:38.911795 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:38.911764    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5495ee2c-12e1-43de-8d3c-1db522e7dc4f" path="/var/lib/kubelet/pods/5495ee2c-12e1-43de-8d3c-1db522e7dc4f/volumes"
Nov 02 22:30:40.543618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:40.543584    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t" podUID="cfa258a7-12ac-4547-80ce-3e4826883324" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:30:41.164372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:41.164337    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"]
Nov 02 22:30:41.260318 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:41.260286    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"]
Nov 02 22:30:44.113837 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:44.113796    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:30:46.164749 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:46.164711    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5" podUID="a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" containerName="payload-processing" containerID="cri-o://47930b1d0f847ab72afb0ea72d0d35695d5141c76c31118fcdddf45cfe231a17" gracePeriod=25
Nov 02 22:30:46.261347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:46.261312    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl" podUID="c090bef4-b276-45ba-9754-834012be133f" containerName="payload-pre-processing" containerID="cri-o://3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c" gracePeriod=25
Nov 02 22:30:46.901225 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:46.901162    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 22:30:47.025869 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.025840    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa258a7-12ac-4547-80ce-3e4826883324-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa258a7-12ac-4547-80ce-3e4826883324-plugins-config-volume\") pod \"cfa258a7-12ac-4547-80ce-3e4826883324\" (UID: \"cfa258a7-12ac-4547-80ce-3e4826883324\") "
Nov 02 22:30:47.026066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.025958    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa258a7-12ac-4547-80ce-3e4826883324-kube-api-access-bb989\" (UniqueName: \"kubernetes.io/projected/cfa258a7-12ac-4547-80ce-3e4826883324-kube-api-access-bb989\") pod \"cfa258a7-12ac-4547-80ce-3e4826883324\" (UID: \"cfa258a7-12ac-4547-80ce-3e4826883324\") "
Nov 02 22:30:47.026276 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.026247    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa258a7-12ac-4547-80ce-3e4826883324-plugins-config-volume" pod "cfa258a7-12ac-4547-80ce-3e4826883324" (UID: "cfa258a7-12ac-4547-80ce-3e4826883324"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:30:47.028053 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.028023    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa258a7-12ac-4547-80ce-3e4826883324-kube-api-access-bb989" pod "cfa258a7-12ac-4547-80ce-3e4826883324" (UID: "cfa258a7-12ac-4547-80ce-3e4826883324"). InnerVolumeSpecName "kube-api-access-bb989". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:30:47.123123 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.123083    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5" podUID="a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:30:47.123292 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.123160    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl" podUID="c090bef4-b276-45ba-9754-834012be133f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:30:47.127604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.127582    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb989\" (UniqueName: \"kubernetes.io/projected/cfa258a7-12ac-4547-80ce-3e4826883324-kube-api-access-bb989\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:47.127665 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.127608    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa258a7-12ac-4547-80ce-3e4826883324-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:30:47.262123 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.262083    4528 generic.go:356] "Generic (PLEG): container finished" podID="cfa258a7-12ac-4547-80ce-3e4826883324" containerID="63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837" exitCode=0
Nov 02 22:30:47.262531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.262149    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"
Nov 02 22:30:47.262531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.262145    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t" event={"ID":"cfa258a7-12ac-4547-80ce-3e4826883324","Type":"ContainerDied","Data":"63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837"}
Nov 02 22:30:47.262531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.262261    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t" event={"ID":"cfa258a7-12ac-4547-80ce-3e4826883324","Type":"ContainerDied","Data":"6860ce65d3f71437fa11bba9846d05f19ed34cf9f86aeea58df227ac33c4381e"}
Nov 02 22:30:47.262531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.262281    4528 scope.go:122] "RemoveContainer" containerID="63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837"
Nov 02 22:30:47.271785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.271762    4528 scope.go:122] "RemoveContainer" containerID="63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837"
Nov 02 22:30:47.272052 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:30:47.272033    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837\": container with ID starting with 63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837 not found: ID does not exist" containerID="63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837"
Nov 02 22:30:47.272136 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.272063    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837"} err="failed to get container status \"63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837\": rpc error: code = NotFound desc = could not find container \"63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837\": container with ID starting with 63fc5c762e0894debe91f3b48e8d029b21dcead0ab617c2cf5d3a587ae037837 not found: ID does not exist"
Nov 02 22:30:47.426952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.426918    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"]
Nov 02 22:30:47.432483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:47.432446    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6e535d-868bbcc497-m6t6t"]
Nov 02 22:30:48.911785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:48.911748    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa258a7-12ac-4547-80ce-3e4826883324" path="/var/lib/kubelet/pods/cfa258a7-12ac-4547-80ce-3e4826883324/volumes"
Nov 02 22:30:49.046737 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:49.046705    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"]
Nov 02 22:30:49.096612 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:49.096585    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1339d374-57kst"]
Nov 02 22:30:50.912571 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:50.912538    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0099729c-ec0f-4b6b-b375-0b743fd0e19c" path="/var/lib/kubelet/pods/0099729c-ec0f-4b6b-b375-0b743fd0e19c/volumes"
Nov 02 22:30:57.123299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:57.123262    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl" podUID="c090bef4-b276-45ba-9754-834012be133f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:30:57.123299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:57.123278    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5" podUID="a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:30:58.046975 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:58.046943    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"]
Nov 02 22:30:58.097333 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:58.097305    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa13b47-gz9gc"]
Nov 02 22:30:58.911900 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:30:58.911855    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d047fc-f926-4100-9ac2-d1a5ea5bc266" path="/var/lib/kubelet/pods/c4d047fc-f926-4100-9ac2-d1a5ea5bc266/volumes"
Nov 02 22:31:06.374441 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.374411    4528 generic.go:356] "Generic (PLEG): container finished" podID="a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" containerID="47930b1d0f847ab72afb0ea72d0d35695d5141c76c31118fcdddf45cfe231a17" exitCode=0
Nov 02 22:31:06.374887 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.374495    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5" event={"ID":"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5","Type":"ContainerDied","Data":"47930b1d0f847ab72afb0ea72d0d35695d5141c76c31118fcdddf45cfe231a17"}
Nov 02 22:31:06.510686 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.510665    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 22:31:06.515735 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.515714    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-kube-api-access-gxsl8\" (UniqueName: \"kubernetes.io/projected/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-kube-api-access-gxsl8\") pod \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\" (UID: \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\") "
Nov 02 22:31:06.515837 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.515758    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-plugins-config-volume\") pod \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\" (UID: \"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5\") "
Nov 02 22:31:06.516131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.516097    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-plugins-config-volume" pod "a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" (UID: "a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:31:06.517730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.517703    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-kube-api-access-gxsl8" pod "a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" (UID: "a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5"). InnerVolumeSpecName "kube-api-access-gxsl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:06.530962 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.530944    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 22:31:06.616688 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.616658    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c090bef4-b276-45ba-9754-834012be133f-kube-api-access-5972j\" (UniqueName: \"kubernetes.io/projected/c090bef4-b276-45ba-9754-834012be133f-kube-api-access-5972j\") pod \"c090bef4-b276-45ba-9754-834012be133f\" (UID: \"c090bef4-b276-45ba-9754-834012be133f\") "
Nov 02 22:31:06.616848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.616776    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c090bef4-b276-45ba-9754-834012be133f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c090bef4-b276-45ba-9754-834012be133f-plugins-config-volume\") pod \"c090bef4-b276-45ba-9754-834012be133f\" (UID: \"c090bef4-b276-45ba-9754-834012be133f\") "
Nov 02 22:31:06.617055 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.617036    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxsl8\" (UniqueName: \"kubernetes.io/projected/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-kube-api-access-gxsl8\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:06.617149 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.617062    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:06.617205 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.617178    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c090bef4-b276-45ba-9754-834012be133f-plugins-config-volume" pod "c090bef4-b276-45ba-9754-834012be133f" (UID: "c090bef4-b276-45ba-9754-834012be133f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:31:06.618746 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.618724    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c090bef4-b276-45ba-9754-834012be133f-kube-api-access-5972j" pod "c090bef4-b276-45ba-9754-834012be133f" (UID: "c090bef4-b276-45ba-9754-834012be133f"). InnerVolumeSpecName "kube-api-access-5972j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:06.717646 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.717568    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c090bef4-b276-45ba-9754-834012be133f-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:06.717646 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:06.717594    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5972j\" (UniqueName: \"kubernetes.io/projected/c090bef4-b276-45ba-9754-834012be133f-kube-api-access-5972j\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:07.381671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.381634    4528 generic.go:356] "Generic (PLEG): container finished" podID="c090bef4-b276-45ba-9754-834012be133f" containerID="3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c" exitCode=0
Nov 02 22:31:07.382159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.381684    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"
Nov 02 22:31:07.382159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.381727    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl" event={"ID":"c090bef4-b276-45ba-9754-834012be133f","Type":"ContainerDied","Data":"3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c"}
Nov 02 22:31:07.382159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.381779    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl" event={"ID":"c090bef4-b276-45ba-9754-834012be133f","Type":"ContainerDied","Data":"245204a3f1002bc4a8c810084058e94d06cc1fba56c920748498006b31ff0eab"}
Nov 02 22:31:07.382159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.381807    4528 scope.go:122] "RemoveContainer" containerID="3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c"
Nov 02 22:31:07.383203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.383185    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"
Nov 02 22:31:07.383298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.383184    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5" event={"ID":"a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5","Type":"ContainerDied","Data":"e0be525b83ea1f7a67259c209ada420d75c01179f81efe2f8d5a1c2747bf7a70"}
Nov 02 22:31:07.391006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.390988    4528 scope.go:122] "RemoveContainer" containerID="3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c"
Nov 02 22:31:07.391278 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:31:07.391258    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c\": container with ID starting with 3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c not found: ID does not exist" containerID="3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c"
Nov 02 22:31:07.391336 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.391285    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c"} err="failed to get container status \"3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c\": rpc error: code = NotFound desc = could not find container \"3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c\": container with ID starting with 3af6fc0555912b8e3794dee22672c36275634e362a9e3e0c768b011338819e5c not found: ID does not exist"
Nov 02 22:31:07.391336 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.391301    4528 scope.go:122] "RemoveContainer" containerID="47930b1d0f847ab72afb0ea72d0d35695d5141c76c31118fcdddf45cfe231a17"
Nov 02 22:31:07.545453 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.545424    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"]
Nov 02 22:31:07.595239 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.595213    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-0f0874-789d467cb5-8k4s5"]
Nov 02 22:31:07.655203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.655173    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"]
Nov 02 22:31:07.658237 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:07.658211    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0f0874-d48cc8559-92jnl"]
Nov 02 22:31:08.912894 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:08.912861    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" path="/var/lib/kubelet/pods/a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5/volumes"
Nov 02 22:31:08.913307 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:08.913265    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c090bef4-b276-45ba-9754-834012be133f" path="/var/lib/kubelet/pods/c090bef4-b276-45ba-9754-834012be133f/volumes"
Nov 02 22:31:11.046371 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:11.046340    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"]
Nov 02 22:31:11.096524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:11.096497    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea93f8-t997m"]
Nov 02 22:31:12.912087 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:12.912055    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8bc17e7-9227-48c2-8e88-b7631ded64dc" path="/var/lib/kubelet/pods/f8bc17e7-9227-48c2-8e88-b7631ded64dc/volumes"
Nov 02 22:31:14.112998 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:31:14.112965    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:31:16.970095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:16.970062    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:31:20.713011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:20.712981    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"]
Nov 02 22:31:20.713472 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:20.713211    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy" containerID="cri-o://30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5" gracePeriod=30
Nov 02 22:31:21.463689 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.463650    4528 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:31:21.463871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.463720    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:31:21.812568 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812533    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"]
Nov 02 22:31:21.812958 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812933    4528 cpu_manager.go:414] "containerMap: removing container" podUID="22a24acc-dc6f-4ef9-b75c-332992234503" containerName="revoke-keys"
Nov 02 22:31:21.812958 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812945    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a24acc-dc6f-4ef9-b75c-332992234503" containerName="revoke-keys"
Nov 02 22:31:21.812958 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812955    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerName="cleanup"
Nov 02 22:31:21.812958 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812960    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812974    4528 cpu_manager.go:414] "containerMap: removing container" podUID="991fa00a-364a-4544-a478-66e3eac54f46" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812980    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991fa00a-364a-4544-a478-66e3eac54f46" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812987    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812991    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.812998    4528 cpu_manager.go:414] "containerMap: removing container" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813003    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813012    4528 cpu_manager.go:414] "containerMap: removing container" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813016    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813021    4528 cpu_manager.go:414] "containerMap: removing container" podUID="cfa258a7-12ac-4547-80ce-3e4826883324" containerName="payload-processing"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813027    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa258a7-12ac-4547-80ce-3e4826883324" containerName="payload-processing"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813036    4528 cpu_manager.go:414] "containerMap: removing container" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813041    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813049    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" containerName="payload-processing"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813053    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" containerName="payload-processing"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813058    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d921a58d-7bdc-4793-bf83-d53ffb571b0f" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813062    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d921a58d-7bdc-4793-bf83-d53ffb571b0f" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813070    4528 cpu_manager.go:414] "containerMap: removing container" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813074    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813079    4528 cpu_manager.go:414] "containerMap: removing container" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813084    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813090    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c090bef4-b276-45ba-9754-834012be133f" containerName="payload-pre-processing"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813094    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c090bef4-b276-45ba-9754-834012be133f" containerName="payload-pre-processing"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813103    4528 cpu_manager.go:414] "containerMap: removing container" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813119    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813131    4528 cpu_manager.go:414] "containerMap: removing container" podUID="991fa00a-364a-4544-a478-66e3eac54f46" containerName="cleanup"
Nov 02 22:31:21.813129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813135    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991fa00a-364a-4544-a478-66e3eac54f46" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813206    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813213    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813219    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813225    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c090bef4-b276-45ba-9754-834012be133f" containerName="payload-pre-processing"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813232    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813238    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813243    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="991fa00a-364a-4544-a478-66e3eac54f46" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813249    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d921a58d-7bdc-4793-bf83-d53ffb571b0f" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813254    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a24acc-dc6f-4ef9-b75c-332992234503" containerName="revoke-keys"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813260    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="991fa00a-364a-4544-a478-66e3eac54f46" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813264    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerName="cleanup"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813268    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c65c87-4bcb-4a5c-b4d1-396f2a7f3612" containerName="main"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813273    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa258a7-12ac-4547-80ce-3e4826883324" containerName="payload-processing"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813279    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bd8bfc-643e-4750-99c2-d6bbcfa6abf5" containerName="payload-processing"
Nov 02 22:31:21.813928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.813284    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d921a58d-7bdc-4793-bf83-d53ffb571b0f" containerName="cleanup"
Nov 02 22:31:21.816557 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.816541    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:21.867480 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.867443    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7942a61c-wn2rg\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:21.867610 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.867508    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkx2j\" (UniqueName: \"kubernetes.io/projected/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-kube-api-access-lkx2j\") pod \"maas-api-revoke-sub-7942a61c-wn2rg\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:21.867656 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.867603    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7942a61c-wn2rg\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:21.870223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.870200    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:31:21.870396 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.870273    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:31:21.870396 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.870385    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:31:21.870594 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.870502    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:31:21.872757 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.872736    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"]
Nov 02 22:31:21.968615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.968582    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkx2j\" (UniqueName: \"kubernetes.io/projected/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-kube-api-access-lkx2j\") pod \"maas-api-revoke-sub-7942a61c-wn2rg\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:21.968615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.968618    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7942a61c-wn2rg\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:21.968898 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.968874    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7942a61c-wn2rg\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:21.969486 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.969461    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7942a61c-wn2rg\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:21.970925 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:21.970906    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7942a61c-wn2rg\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:22.025259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:22.025238    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkx2j\" (UniqueName: \"kubernetes.io/projected/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-kube-api-access-lkx2j\") pod \"maas-api-revoke-sub-7942a61c-wn2rg\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:22.131202 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:22.131136    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:22.496173 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:31:22.496141    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15617ad_4b5e_4693_9f9b_ee3ee8e674cc.slice/crio-93937d3e80fc07ed5cb4d7a348eb61fb95669a8f5a950ffcdba64a55e2e873d2 WatchSource:0}: Error finding container 93937d3e80fc07ed5cb4d7a348eb61fb95669a8f5a950ffcdba64a55e2e873d2: Status 404 returned error can't find the container with id 93937d3e80fc07ed5cb4d7a348eb61fb95669a8f5a950ffcdba64a55e2e873d2
Nov 02 22:31:22.498585 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:22.498561    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"]
Nov 02 22:31:23.471619 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:23.471584    4528 generic.go:356] "Generic (PLEG): container finished" podID="d15617ad-4b5e-4693-9f9b-ee3ee8e674cc" containerID="95f96cc1028913c9ace1ad951eee4090999ddaa95acf3e7fbff547c9dc0cbef1" exitCode=0
Nov 02 22:31:23.471977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:23.471626    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg" event={"ID":"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc","Type":"ContainerDied","Data":"95f96cc1028913c9ace1ad951eee4090999ddaa95acf3e7fbff547c9dc0cbef1"}
Nov 02 22:31:23.471977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:23.471714    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg" event={"ID":"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc","Type":"ContainerStarted","Data":"93937d3e80fc07ed5cb4d7a348eb61fb95669a8f5a950ffcdba64a55e2e873d2"}
Nov 02 22:31:24.605801 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.605778    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:24.697003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.696981    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-kube-api-access-lkx2j\" (UniqueName: \"kubernetes.io/projected/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-kube-api-access-lkx2j\") pod \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") "
Nov 02 22:31:24.697169 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.697021    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-xks-ca\") pod \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") "
Nov 02 22:31:24.697169 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.697083    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-openshift-ca\") pod \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\" (UID: \"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc\") "
Nov 02 22:31:24.697484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.697464    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-openshift-ca" pod "d15617ad-4b5e-4693-9f9b-ee3ee8e674cc" (UID: "d15617ad-4b5e-4693-9f9b-ee3ee8e674cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:31:24.699054 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.699035    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-xks-ca" pod "d15617ad-4b5e-4693-9f9b-ee3ee8e674cc" (UID: "d15617ad-4b5e-4693-9f9b-ee3ee8e674cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:31:24.699158 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.699050    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-kube-api-access-lkx2j" pod "d15617ad-4b5e-4693-9f9b-ee3ee8e674cc" (UID: "d15617ad-4b5e-4693-9f9b-ee3ee8e674cc"). InnerVolumeSpecName "kube-api-access-lkx2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:24.798291 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.798261    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:24.798291 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.798286    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkx2j\" (UniqueName: \"kubernetes.io/projected/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-kube-api-access-lkx2j\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:24.798483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:24.798298    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:25.483825 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:25.483791    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"
Nov 02 22:31:25.484032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:25.483790    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg" event={"ID":"d15617ad-4b5e-4693-9f9b-ee3ee8e674cc","Type":"ContainerDied","Data":"93937d3e80fc07ed5cb4d7a348eb61fb95669a8f5a950ffcdba64a55e2e873d2"}
Nov 02 22:31:25.484032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:25.483910    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93937d3e80fc07ed5cb4d7a348eb61fb95669a8f5a950ffcdba64a55e2e873d2"
Nov 02 22:31:25.969822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:25.969797    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 22:31:26.110892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.110863    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-kube-api-access-4hvnd\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-kube-api-access-4hvnd\") pod \"387be747-622a-4118-89c1-9565f9c5e947\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") "
Nov 02 22:31:26.111095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.110898    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-istio-token\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-istio-token\") pod \"387be747-622a-4118-89c1-9565f9c5e947\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") "
Nov 02 22:31:26.111095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.110922    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-certs\") pod \"387be747-622a-4118-89c1-9565f9c5e947\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") "
Nov 02 22:31:26.111095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.110941    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-data\") pod \"387be747-622a-4118-89c1-9565f9c5e947\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") "
Nov 02 22:31:26.111095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.110974    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/387be747-622a-4118-89c1-9565f9c5e947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387be747-622a-4118-89c1-9565f9c5e947-istio-podinfo\") pod \"387be747-622a-4118-89c1-9565f9c5e947\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") "
Nov 02 22:31:26.111095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111006    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387be747-622a-4118-89c1-9565f9c5e947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387be747-622a-4118-89c1-9565f9c5e947-istiod-ca-cert\") pod \"387be747-622a-4118-89c1-9565f9c5e947\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") "
Nov 02 22:31:26.111095 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111074    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-socket\") pod \"387be747-622a-4118-89c1-9565f9c5e947\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") "
Nov 02 22:31:26.111481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111148    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-envoy\") pod \"387be747-622a-4118-89c1-9565f9c5e947\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") "
Nov 02 22:31:26.111481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111193    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-credential-socket\") pod \"387be747-622a-4118-89c1-9565f9c5e947\" (UID: \"387be747-622a-4118-89c1-9565f9c5e947\") "
Nov 02 22:31:26.111481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111201    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-certs" pod "387be747-622a-4118-89c1-9565f9c5e947" (UID: "387be747-622a-4118-89c1-9565f9c5e947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:26.111481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111326    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-data" pod "387be747-622a-4118-89c1-9565f9c5e947" (UID: "387be747-622a-4118-89c1-9565f9c5e947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:26.111481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111425    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-socket" pod "387be747-622a-4118-89c1-9565f9c5e947" (UID: "387be747-622a-4118-89c1-9565f9c5e947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:26.111771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111620    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:26.111771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111642    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:26.111771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111655    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:26.111771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111624    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-credential-socket" pod "387be747-622a-4118-89c1-9565f9c5e947" (UID: "387be747-622a-4118-89c1-9565f9c5e947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:26.111771 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.111727    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387be747-622a-4118-89c1-9565f9c5e947-istiod-ca-cert" pod "387be747-622a-4118-89c1-9565f9c5e947" (UID: "387be747-622a-4118-89c1-9565f9c5e947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:31:26.113660 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.113636    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/387be747-622a-4118-89c1-9565f9c5e947-istio-podinfo" pod "387be747-622a-4118-89c1-9565f9c5e947" (UID: "387be747-622a-4118-89c1-9565f9c5e947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:31:26.113772 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.113706    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-kube-api-access-4hvnd" pod "387be747-622a-4118-89c1-9565f9c5e947" (UID: "387be747-622a-4118-89c1-9565f9c5e947"). InnerVolumeSpecName "kube-api-access-4hvnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:26.113772 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.113715    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-envoy" pod "387be747-622a-4118-89c1-9565f9c5e947" (UID: "387be747-622a-4118-89c1-9565f9c5e947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:26.113928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.113904    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-istio-token" pod "387be747-622a-4118-89c1-9565f9c5e947" (UID: "387be747-622a-4118-89c1-9565f9c5e947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:26.123826 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.123796    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"]
Nov 02 22:31:26.128585 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.128565    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"]
Nov 02 22:31:26.212793 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.212762    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hvnd\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-kube-api-access-4hvnd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:26.212793 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.212788    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387be747-622a-4118-89c1-9565f9c5e947-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:26.212793 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.212798    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387be747-622a-4118-89c1-9565f9c5e947-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:26.213056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.212806    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387be747-622a-4118-89c1-9565f9c5e947-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:26.213056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.212814    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:26.213056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.212824    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387be747-622a-4118-89c1-9565f9c5e947-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:26.490882 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.490795    4528 generic.go:356] "Generic (PLEG): container finished" podID="387be747-622a-4118-89c1-9565f9c5e947" containerID="30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5" exitCode=0
Nov 02 22:31:26.490882 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.490843    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"
Nov 02 22:31:26.491077 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.490883    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" event={"ID":"387be747-622a-4118-89c1-9565f9c5e947","Type":"ContainerDied","Data":"30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5"}
Nov 02 22:31:26.491077 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.490939    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x" event={"ID":"387be747-622a-4118-89c1-9565f9c5e947","Type":"ContainerDied","Data":"ae8b0970a7701a36db8d8b0973262580cefcf92e4eabc90ba6a4f9d62847814d"}
Nov 02 22:31:26.491077 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.490964    4528 scope.go:122] "RemoveContainer" containerID="30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5"
Nov 02 22:31:26.501145 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.501127    4528 scope.go:122] "RemoveContainer" containerID="37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1"
Nov 02 22:31:26.509823 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.509802    4528 scope.go:122] "RemoveContainer" containerID="30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5"
Nov 02 22:31:26.510077 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:31:26.510056    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5\": container with ID starting with 30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5 not found: ID does not exist" containerID="30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5"
Nov 02 22:31:26.510176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.510086    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5"} err="failed to get container status \"30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5\": rpc error: code = NotFound desc = could not find container \"30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5\": container with ID starting with 30a3d14953983bb94a81b01df45fef1348a585db32a00bda7e752f430c7f92f5 not found: ID does not exist"
Nov 02 22:31:26.510176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.510104    4528 scope.go:122] "RemoveContainer" containerID="37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1"
Nov 02 22:31:26.510365 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:31:26.510348    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1\": container with ID starting with 37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1 not found: ID does not exist" containerID="37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1"
Nov 02 22:31:26.510433 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.510368    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1"} err="failed to get container status \"37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1\": rpc error: code = NotFound desc = could not find container \"37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1\": container with ID starting with 37560e00a3c34a84f6e2d4530fc6ecd69a04417e8ea1e5dad4444f1d1739eea1 not found: ID does not exist"
Nov 02 22:31:26.659162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.659123    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"]
Nov 02 22:31:26.661945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.661911    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-6j86x"]
Nov 02 22:31:26.912262 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:26.912229    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387be747-622a-4118-89c1-9565f9c5e947" path="/var/lib/kubelet/pods/387be747-622a-4118-89c1-9565f9c5e947/volumes"
Nov 02 22:31:33.246969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.246884    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"]
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247314    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerName="cleanup"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247327    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerName="cleanup"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247339    4528 cpu_manager.go:414] "containerMap: removing container" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247344    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247356    4528 cpu_manager.go:414] "containerMap: removing container" podUID="991fa00a-364a-4544-a478-66e3eac54f46" containerName="cleanup"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247361    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991fa00a-364a-4544-a478-66e3eac54f46" containerName="cleanup"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247369    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d15617ad-4b5e-4693-9f9b-ee3ee8e674cc" containerName="revoke-keys"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247376    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15617ad-4b5e-4693-9f9b-ee3ee8e674cc" containerName="revoke-keys"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247391    4528 cpu_manager.go:414] "containerMap: removing container" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerName="cleanup"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247396    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerName="cleanup"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247403    4528 cpu_manager.go:414] "containerMap: removing container" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247407    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247418    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d921a58d-7bdc-4793-bf83-d53ffb571b0f" containerName="cleanup"
Nov 02 22:31:33.247444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247422    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d921a58d-7bdc-4793-bf83-d53ffb571b0f" containerName="cleanup"
Nov 02 22:31:33.248156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247493    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa1bf8f-c68e-421e-9b92-b5c49a6c96da" containerName="cleanup"
Nov 02 22:31:33.248156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247504    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc251060-916a-4354-9ad9-35a9fd77fada" containerName="cleanup"
Nov 02 22:31:33.248156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247512    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy"
Nov 02 22:31:33.248156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247518    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="991fa00a-364a-4544-a478-66e3eac54f46" containerName="cleanup"
Nov 02 22:31:33.248156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247524    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cff92e-d2fa-4e9d-8275-1d9e401af54c" containerName="cleanup"
Nov 02 22:31:33.248156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247529    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15617ad-4b5e-4693-9f9b-ee3ee8e674cc" containerName="revoke-keys"
Nov 02 22:31:33.248156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.247536    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="387be747-622a-4118-89c1-9565f9c5e947" containerName="istio-proxy"
Nov 02 22:31:33.252231 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.252210    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.305501 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.305454    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:31:33.320342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.320312    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"]
Nov 02 22:31:33.355842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.355814    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:31:33.355842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.355821    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:31:33.356016 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.355881    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:31:33.388362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.388335    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgpc\" (UniqueName: \"kubernetes.io/projected/ae4a91b7-27ab-4795-a2c7-14816da0e206-kube-api-access-kjgpc\") pod \"maas-api-revoke-sub-379f854e-h7tms\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.388496 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.388390    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-379f854e-h7tms\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.388594 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.388574    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-379f854e-h7tms\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.489801 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.489771    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-379f854e-h7tms\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.489955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.489827    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-379f854e-h7tms\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.489955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.489878    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgpc\" (UniqueName: \"kubernetes.io/projected/ae4a91b7-27ab-4795-a2c7-14816da0e206-kube-api-access-kjgpc\") pod \"maas-api-revoke-sub-379f854e-h7tms\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.490584 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.490565    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-379f854e-h7tms\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.491993 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.491974    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-379f854e-h7tms\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.546577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.546550    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgpc\" (UniqueName: \"kubernetes.io/projected/ae4a91b7-27ab-4795-a2c7-14816da0e206-kube-api-access-kjgpc\") pod \"maas-api-revoke-sub-379f854e-h7tms\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.565526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.565504    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:33.931403 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:31:33.931365    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4a91b7_27ab_4795_a2c7_14816da0e206.slice/crio-d24168192d7f97a56ead80a53d81e9d44217a57b930e6cc586b891dc1befcfec WatchSource:0}: Error finding container d24168192d7f97a56ead80a53d81e9d44217a57b930e6cc586b891dc1befcfec: Status 404 returned error can't find the container with id d24168192d7f97a56ead80a53d81e9d44217a57b930e6cc586b891dc1befcfec
Nov 02 22:31:33.934099 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:33.934076    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"]
Nov 02 22:31:34.535040 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:34.535006    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms" event={"ID":"ae4a91b7-27ab-4795-a2c7-14816da0e206","Type":"ContainerStarted","Data":"31245f800c6e2577058a18af1713b4dfcc85c13f37ec1dfddc5615f077e709d8"}
Nov 02 22:31:34.535461 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:34.535060    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms" event={"ID":"ae4a91b7-27ab-4795-a2c7-14816da0e206","Type":"ContainerStarted","Data":"d24168192d7f97a56ead80a53d81e9d44217a57b930e6cc586b891dc1befcfec"}
Nov 02 22:31:35.542930 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:35.542892    4528 generic.go:356] "Generic (PLEG): container finished" podID="ae4a91b7-27ab-4795-a2c7-14816da0e206" containerID="31245f800c6e2577058a18af1713b4dfcc85c13f37ec1dfddc5615f077e709d8" exitCode=0
Nov 02 22:31:35.543417 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:35.542983    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms" event={"ID":"ae4a91b7-27ab-4795-a2c7-14816da0e206","Type":"ContainerDied","Data":"31245f800c6e2577058a18af1713b4dfcc85c13f37ec1dfddc5615f077e709d8"}
Nov 02 22:31:36.680491 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.680470    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:36.752993 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.752965    4528 scope.go:122] "RemoveContainer" containerID="a3420ab7db5e937b53ff27cc1bdfbc18f95ae04086b8a5a86e922e5514539841"
Nov 02 22:31:36.821497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.821433    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4a91b7-27ab-4795-a2c7-14816da0e206-kube-api-access-kjgpc\" (UniqueName: \"kubernetes.io/projected/ae4a91b7-27ab-4795-a2c7-14816da0e206-kube-api-access-kjgpc\") pod \"ae4a91b7-27ab-4795-a2c7-14816da0e206\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") "
Nov 02 22:31:36.821497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.821473    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-openshift-ca\") pod \"ae4a91b7-27ab-4795-a2c7-14816da0e206\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") "
Nov 02 22:31:36.821713 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.821506    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-xks-ca\") pod \"ae4a91b7-27ab-4795-a2c7-14816da0e206\" (UID: \"ae4a91b7-27ab-4795-a2c7-14816da0e206\") "
Nov 02 22:31:36.821857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.821835    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-openshift-ca" pod "ae4a91b7-27ab-4795-a2c7-14816da0e206" (UID: "ae4a91b7-27ab-4795-a2c7-14816da0e206"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:31:36.823526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.823503    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4a91b7-27ab-4795-a2c7-14816da0e206-kube-api-access-kjgpc" pod "ae4a91b7-27ab-4795-a2c7-14816da0e206" (UID: "ae4a91b7-27ab-4795-a2c7-14816da0e206"). InnerVolumeSpecName "kube-api-access-kjgpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:36.823613 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.823542    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-xks-ca" pod "ae4a91b7-27ab-4795-a2c7-14816da0e206" (UID: "ae4a91b7-27ab-4795-a2c7-14816da0e206"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:31:36.922547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.922521    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjgpc\" (UniqueName: \"kubernetes.io/projected/ae4a91b7-27ab-4795-a2c7-14816da0e206-kube-api-access-kjgpc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:36.922547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.922547    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:36.922699 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:36.922557    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a91b7-27ab-4795-a2c7-14816da0e206-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:37.555785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:37.555748    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"
Nov 02 22:31:37.555937 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:37.555747    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms" event={"ID":"ae4a91b7-27ab-4795-a2c7-14816da0e206","Type":"ContainerDied","Data":"d24168192d7f97a56ead80a53d81e9d44217a57b930e6cc586b891dc1befcfec"}
Nov 02 22:31:37.555937 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:37.555881    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d24168192d7f97a56ead80a53d81e9d44217a57b930e6cc586b891dc1befcfec"
Nov 02 22:31:39.626333 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:39.626296    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:31:41.143174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.143094    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" containerID="cri-o://143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449" gracePeriod=45
Nov 02 22:31:41.146865 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.146814    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main" containerID="cri-o://8c6ae959acbb24a3a65d09dc1c070d707433117f6f6bf5bac3f17afaa453c25e" gracePeriod=45
Nov 02 22:31:41.466390 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.466040    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 22:31:41.483889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.482627    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-model-cache\") pod \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") "
Nov 02 22:31:41.483889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.482679    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-dshm\") pod \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") "
Nov 02 22:31:41.483889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.482713    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tls-certs\") pod \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") "
Nov 02 22:31:41.483889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.482826    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tmp-dir\") pod \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") "
Nov 02 22:31:41.483889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.482888    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-home\") pod \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") "
Nov 02 22:31:41.483889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.482917    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-kube-api-access-45n5n\" (UniqueName: \"kubernetes.io/projected/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-kube-api-access-45n5n\") pod \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\" (UID: \"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d\") "
Nov 02 22:31:41.495997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.495369    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tmp-dir" pod "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" (UID: "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:41.495997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.495560    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-home" pod "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" (UID: "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:41.495997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.495734    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-model-cache" pod "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" (UID: "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:41.539665 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.539607    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tls-certs" pod "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" (UID: "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:31:41.569764 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.569715    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-kube-api-access-45n5n" pod "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" (UID: "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d"). InnerVolumeSpecName "kube-api-access-45n5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:41.591132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.583956    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.591132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.583988    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.591132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.584004    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.591132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.584016    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.591132 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.584029    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45n5n\" (UniqueName: \"kubernetes.io/projected/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-kube-api-access-45n5n\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.593284 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.591633    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-dshm" pod "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" (UID: "2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:41.608588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.608418    4528 generic.go:356] "Generic (PLEG): container finished" podID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerID="8c6ae959acbb24a3a65d09dc1c070d707433117f6f6bf5bac3f17afaa453c25e" exitCode=0
Nov 02 22:31:41.608588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.608499    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" event={"ID":"4dde1ed2-c163-4938-a9f2-46f39a4d257a","Type":"ContainerDied","Data":"8c6ae959acbb24a3a65d09dc1c070d707433117f6f6bf5bac3f17afaa453c25e"}
Nov 02 22:31:41.608588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.608534    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt" event={"ID":"4dde1ed2-c163-4938-a9f2-46f39a4d257a","Type":"ContainerDied","Data":"0e6ac341f6c13a756253f090c523fcab3cf48b92b51274d9ffd55d7c3f3778d9"}
Nov 02 22:31:41.608588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.608552    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6ac341f6c13a756253f090c523fcab3cf48b92b51274d9ffd55d7c3f3778d9"
Nov 02 22:31:41.619208 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.618911    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 22:31:41.619807 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.619784    4528 generic.go:356] "Generic (PLEG): container finished" podID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerID="143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449" exitCode=0
Nov 02 22:31:41.620016 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.619986    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" event={"ID":"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d","Type":"ContainerDied","Data":"143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449"}
Nov 02 22:31:41.620222 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.620207    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" event={"ID":"2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d","Type":"ContainerDied","Data":"cf344fc56d9372554e81c70f177ac4f46b1e0f0422c0efe2f3a6f9deb5029461"}
Nov 02 22:31:41.620361 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.620348    4528 scope.go:122] "RemoveContainer" containerID="143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449"
Nov 02 22:31:41.620639 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.620624    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"
Nov 02 22:31:41.656271 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.656185    4528 scope.go:122] "RemoveContainer" containerID="143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449"
Nov 02 22:31:41.656635 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:31:41.656595    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449\": container with ID starting with 143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449 not found: ID does not exist" containerID="143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449"
Nov 02 22:31:41.656712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.656632    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449"} err="failed to get container status \"143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449\": rpc error: code = NotFound desc = could not find container \"143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449\": container with ID starting with 143fdabebf3447a8910e45e2105d09012cea89d978947b19073061da607d4449 not found: ID does not exist"
Nov 02 22:31:41.684684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.684606    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tmp-dir\") pod \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") "
Nov 02 22:31:41.684684 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.684663    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-model-cache\") pod \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") "
Nov 02 22:31:41.684916 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.684694    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dde1ed2-c163-4938-a9f2-46f39a4d257a-kube-api-access-vvnmd\" (UniqueName: \"kubernetes.io/projected/4dde1ed2-c163-4938-a9f2-46f39a4d257a-kube-api-access-vvnmd\") pod \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") "
Nov 02 22:31:41.684916 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.684750    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tls-certs\" (UniqueName: \"kubernetes.io/secret/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tls-certs\") pod \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") "
Nov 02 22:31:41.684916 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.684817    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-home\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-home\") pod \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") "
Nov 02 22:31:41.685082 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.684957    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tmp-dir" pod "4dde1ed2-c163-4938-a9f2-46f39a4d257a" (UID: "4dde1ed2-c163-4938-a9f2-46f39a4d257a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:41.685082 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.684969    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-model-cache" pod "4dde1ed2-c163-4938-a9f2-46f39a4d257a" (UID: "4dde1ed2-c163-4938-a9f2-46f39a4d257a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:41.685415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.685184    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-home" pod "4dde1ed2-c163-4938-a9f2-46f39a4d257a" (UID: "4dde1ed2-c163-4938-a9f2-46f39a4d257a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:41.685415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.685192    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.685415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.685229    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.685415 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.685244    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.687122 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.687083    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tls-certs" pod "4dde1ed2-c163-4938-a9f2-46f39a4d257a" (UID: "4dde1ed2-c163-4938-a9f2-46f39a4d257a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:31:41.687531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.687506    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dde1ed2-c163-4938-a9f2-46f39a4d257a-kube-api-access-vvnmd" pod "4dde1ed2-c163-4938-a9f2-46f39a4d257a" (UID: "4dde1ed2-c163-4938-a9f2-46f39a4d257a"). InnerVolumeSpecName "kube-api-access-vvnmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:31:41.785573 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.785543    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-dshm\") pod \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\" (UID: \"4dde1ed2-c163-4938-a9f2-46f39a4d257a\") "
Nov 02 22:31:41.785785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.785773    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvnmd\" (UniqueName: \"kubernetes.io/projected/4dde1ed2-c163-4938-a9f2-46f39a4d257a-kube-api-access-vvnmd\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.785830 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.785787    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dde1ed2-c163-4938-a9f2-46f39a4d257a-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.785830 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.785796    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:41.787529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.787497    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-dshm" pod "4dde1ed2-c163-4938-a9f2-46f39a4d257a" (UID: "4dde1ed2-c163-4938-a9f2-46f39a4d257a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:31:41.844811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.844770    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"]
Nov 02 22:31:41.848362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.848335    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2"]
Nov 02 22:31:41.886724 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:41.886690    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dde1ed2-c163-4938-a9f2-46f39a4d257a-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:31:42.397309 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:42.397266    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/distinct-2-w0-0c9014-kserve-9d8f684cf-22zg2" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:31:42.627162 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:42.627123    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"
Nov 02 22:31:42.791780 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:42.791745    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"]
Nov 02 22:31:42.793831 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:42.793777    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/distinct-w0-0c9014-kserve-55c58d586d-qt9lt"]
Nov 02 22:31:42.913626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:42.913550    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" path="/var/lib/kubelet/pods/2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d/volumes"
Nov 02 22:31:42.913950 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:31:42.913937    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" path="/var/lib/kubelet/pods/4dde1ed2-c163-4938-a9f2-46f39a4d257a/volumes"
Nov 02 22:31:44.112017 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:31:44.111983    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:32:00.049150 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:00.049101    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"]
Nov 02 22:32:00.098887 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:00.098850    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a4f635e-f8n8q"]
Nov 02 22:32:00.912478 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:00.912442    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a570bd-bef8-4a8c-a49a-0615dfe1bddf" path="/var/lib/kubelet/pods/55a570bd-bef8-4a8c-a49a-0615dfe1bddf/volumes"
Nov 02 22:32:03.059001 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.058963    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"]
Nov 02 22:32:03.059399 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.059370    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main"
Nov 02 22:32:03.059399 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.059381    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main"
Nov 02 22:32:03.059399 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.059392    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main"
Nov 02 22:32:03.059399 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.059398    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main"
Nov 02 22:32:03.059540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.059408    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ae4a91b7-27ab-4795-a2c7-14816da0e206" containerName="revoke-keys"
Nov 02 22:32:03.059540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.059414    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4a91b7-27ab-4795-a2c7-14816da0e206" containerName="revoke-keys"
Nov 02 22:32:03.059540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.059486    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dde1ed2-c163-4938-a9f2-46f39a4d257a" containerName="main"
Nov 02 22:32:03.059540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.059493    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4e9fc7-34d3-4d98-87f5-9faf1efd4f0d" containerName="main"
Nov 02 22:32:03.059540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.059499    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4a91b7-27ab-4795-a2c7-14816da0e206" containerName="revoke-keys"
Nov 02 22:32:03.062834 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.062819    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.148362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.148330    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0c9014-openshift-default-dockercfg-7ll9g\""
Nov 02 22:32:03.169048 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.169013    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"]
Nov 02 22:32:03.183323 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.183289    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.183471 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.183349    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.183471 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.183391    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.183471 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.183424    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.183471 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.183468    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.183618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.183516    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d0cfff-8231-40cf-97e2-75e478b35e53-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.183618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.183542    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh45z\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-kube-api-access-zh45z\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.183618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.183577    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.183798 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.183645    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.284514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.284480    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.284718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.284535    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.284718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.284564    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.284718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.284593    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.284718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.284625    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.284718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.284662    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.284718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.284706    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d0cfff-8231-40cf-97e2-75e478b35e53-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.285078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.284782    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh45z\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-kube-api-access-zh45z\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.285078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.284812    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.285078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.285056    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.285284 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.285206    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.285284 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.285255    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.285374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.285278    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.285617 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.285600    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d0cfff-8231-40cf-97e2-75e478b35e53-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.287380 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.287354    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.287514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.287489    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.341696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.341635    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.341810 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.341717    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh45z\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-kube-api-access-zh45z\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.375796 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.375769    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:03.790721 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:32:03.790688    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d0cfff_8231_40cf_97e2_75e478b35e53.slice/crio-caf36b578837dd4f5517ee11fc99925248bc42dd56316b6f5212563550bba174 WatchSource:0}: Error finding container caf36b578837dd4f5517ee11fc99925248bc42dd56316b6f5212563550bba174: Status 404 returned error can't find the container with id caf36b578837dd4f5517ee11fc99925248bc42dd56316b6f5212563550bba174
Nov 02 22:32:03.793024 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.792994    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:32:03.793129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.793056    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:32:03.793129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.793085    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:32:03.793129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:03.793090    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"]
Nov 02 22:32:04.743435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:04.743398    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" event={"ID":"65d0cfff-8231-40cf-97e2-75e478b35e53","Type":"ContainerStarted","Data":"06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36"}
Nov 02 22:32:04.743865 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:04.743453    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" event={"ID":"65d0cfff-8231-40cf-97e2-75e478b35e53","Type":"ContainerStarted","Data":"caf36b578837dd4f5517ee11fc99925248bc42dd56316b6f5212563550bba174"}
Nov 02 22:32:05.376101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:05.376061    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:05.377758 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:05.377733    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" start-of-body=
Nov 02 22:32:05.377889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:05.377781    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" podUID="65d0cfff-8231-40cf-97e2-75e478b35e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused"
Nov 02 22:32:06.376411 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:06.376377    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" start-of-body=
Nov 02 22:32:06.376811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:06.376428    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" podUID="65d0cfff-8231-40cf-97e2-75e478b35e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused"
Nov 02 22:32:07.376442 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:07.376404    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" start-of-body=
Nov 02 22:32:07.376844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:07.376454    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" podUID="65d0cfff-8231-40cf-97e2-75e478b35e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused"
Nov 02 22:32:08.288068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:08.288010    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" podStartSLOduration=5.287995364 podStartE2EDuration="5.287995364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:04.959819584 +0000 UTC m=+5012.223355901" watchObservedRunningTime="2026-11-02 22:32:08.287995364 +0000 UTC m=+5015.551531681"
Nov 02 22:32:08.289566 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:08.289546    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:32:08.377196 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:08.377163    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" start-of-body=
Nov 02 22:32:08.377556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:08.377216    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" podUID="65d0cfff-8231-40cf-97e2-75e478b35e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused"
Nov 02 22:32:08.423133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:08.402962    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"]
Nov 02 22:32:08.423133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:08.403186    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" podUID="65d0cfff-8231-40cf-97e2-75e478b35e53" containerName="istio-proxy" containerID="cri-o://06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36" gracePeriod=30
Nov 02 22:32:10.049938 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:10.049904    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"]
Nov 02 22:32:10.098547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:10.098518    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461995c1-ndfgp"]
Nov 02 22:32:10.912184 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:10.912152    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb6bac5-7211-44ed-9f19-e159c4037568" path="/var/lib/kubelet/pods/eeb6bac5-7211-44ed-9f19-e159c4037568/volumes"
Nov 02 22:32:14.113843 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:32:14.113805    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:32:15.571244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.571220    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:15.631732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.631705    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-token\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-token\") pod \"65d0cfff-8231-40cf-97e2-75e478b35e53\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") "
Nov 02 22:32:15.631876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.631753    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-envoy\") pod \"65d0cfff-8231-40cf-97e2-75e478b35e53\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") "
Nov 02 22:32:15.631876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.631777    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-data\") pod \"65d0cfff-8231-40cf-97e2-75e478b35e53\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") "
Nov 02 22:32:15.631876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.631798    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-socket\") pod \"65d0cfff-8231-40cf-97e2-75e478b35e53\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") "
Nov 02 22:32:15.631876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.631831    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-kube-api-access-zh45z\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-kube-api-access-zh45z\") pod \"65d0cfff-8231-40cf-97e2-75e478b35e53\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") "
Nov 02 22:32:15.631876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.631855    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-podinfo\") pod \"65d0cfff-8231-40cf-97e2-75e478b35e53\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") "
Nov 02 22:32:15.632187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.631915    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d0cfff-8231-40cf-97e2-75e478b35e53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d0cfff-8231-40cf-97e2-75e478b35e53-istiod-ca-cert\") pod \"65d0cfff-8231-40cf-97e2-75e478b35e53\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") "
Nov 02 22:32:15.632187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.631972    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-certs\") pod \"65d0cfff-8231-40cf-97e2-75e478b35e53\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") "
Nov 02 22:32:15.632187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.632013    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-credential-socket\") pod \"65d0cfff-8231-40cf-97e2-75e478b35e53\" (UID: \"65d0cfff-8231-40cf-97e2-75e478b35e53\") "
Nov 02 22:32:15.632187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.632094    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-data" pod "65d0cfff-8231-40cf-97e2-75e478b35e53" (UID: "65d0cfff-8231-40cf-97e2-75e478b35e53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:32:15.632187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.632153    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-socket" pod "65d0cfff-8231-40cf-97e2-75e478b35e53" (UID: "65d0cfff-8231-40cf-97e2-75e478b35e53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:32:15.632444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.632383    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:15.632444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.632402    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:15.632526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.632472    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-certs" pod "65d0cfff-8231-40cf-97e2-75e478b35e53" (UID: "65d0cfff-8231-40cf-97e2-75e478b35e53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:32:15.632576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.632539    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-credential-socket" pod "65d0cfff-8231-40cf-97e2-75e478b35e53" (UID: "65d0cfff-8231-40cf-97e2-75e478b35e53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:32:15.632615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.632569    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d0cfff-8231-40cf-97e2-75e478b35e53-istiod-ca-cert" pod "65d0cfff-8231-40cf-97e2-75e478b35e53" (UID: "65d0cfff-8231-40cf-97e2-75e478b35e53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:32:15.633917 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.633898    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-kube-api-access-zh45z" pod "65d0cfff-8231-40cf-97e2-75e478b35e53" (UID: "65d0cfff-8231-40cf-97e2-75e478b35e53"). InnerVolumeSpecName "kube-api-access-zh45z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:32:15.634018 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.633971    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-podinfo" pod "65d0cfff-8231-40cf-97e2-75e478b35e53" (UID: "65d0cfff-8231-40cf-97e2-75e478b35e53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:32:15.634277 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.634256    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-envoy" pod "65d0cfff-8231-40cf-97e2-75e478b35e53" (UID: "65d0cfff-8231-40cf-97e2-75e478b35e53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:32:15.634346 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.634283    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-token" pod "65d0cfff-8231-40cf-97e2-75e478b35e53" (UID: "65d0cfff-8231-40cf-97e2-75e478b35e53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:32:15.733578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.733510    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh45z\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-kube-api-access-zh45z\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:15.733578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.733535    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:15.733578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.733545    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d0cfff-8231-40cf-97e2-75e478b35e53-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:15.733578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.733555    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:15.733578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.733569    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:15.733578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.733579    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:15.733870 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.733589    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d0cfff-8231-40cf-97e2-75e478b35e53-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:15.805505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.805474    4528 generic.go:356] "Generic (PLEG): container finished" podID="65d0cfff-8231-40cf-97e2-75e478b35e53" containerID="06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36" exitCode=0
Nov 02 22:32:15.805683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.805522    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"
Nov 02 22:32:15.805683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.805567    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" event={"ID":"65d0cfff-8231-40cf-97e2-75e478b35e53","Type":"ContainerDied","Data":"06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36"}
Nov 02 22:32:15.805683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.805625    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9" event={"ID":"65d0cfff-8231-40cf-97e2-75e478b35e53","Type":"ContainerDied","Data":"caf36b578837dd4f5517ee11fc99925248bc42dd56316b6f5212563550bba174"}
Nov 02 22:32:15.805683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.805653    4528 scope.go:122] "RemoveContainer" containerID="06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36"
Nov 02 22:32:15.815328 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.815313    4528 scope.go:122] "RemoveContainer" containerID="06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36"
Nov 02 22:32:15.815595 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:32:15.815577    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36\": container with ID starting with 06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36 not found: ID does not exist" containerID="06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36"
Nov 02 22:32:15.815643 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.815602    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36"} err="failed to get container status \"06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36\": rpc error: code = NotFound desc = could not find container \"06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36\": container with ID starting with 06d1256b61cb1c65fdd9de2cc324faef017f325e6123abd471dae6492076fb36 not found: ID does not exist"
Nov 02 22:32:15.980518 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.980480    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"]
Nov 02 22:32:15.983358 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:15.983333    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-pp2r9"]
Nov 02 22:32:16.911434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:16.911399    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d0cfff-8231-40cf-97e2-75e478b35e53" path="/var/lib/kubelet/pods/65d0cfff-8231-40cf-97e2-75e478b35e53/volumes"
Nov 02 22:32:21.640085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.640041    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"]
Nov 02 22:32:21.640630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.640609    4528 cpu_manager.go:414] "containerMap: removing container" podUID="65d0cfff-8231-40cf-97e2-75e478b35e53" containerName="istio-proxy"
Nov 02 22:32:21.640729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.640634    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d0cfff-8231-40cf-97e2-75e478b35e53" containerName="istio-proxy"
Nov 02 22:32:21.640789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.640779    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d0cfff-8231-40cf-97e2-75e478b35e53" containerName="istio-proxy"
Nov 02 22:32:21.643914 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.643894    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.688711 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.688683    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b39e1aca-jvd6w\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.688842 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.688743    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69clh\" (UniqueName: \"kubernetes.io/projected/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-kube-api-access-69clh\") pod \"maas-api-revoke-sub-b39e1aca-jvd6w\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.688918 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.688897    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b39e1aca-jvd6w\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.748120 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.748086    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:32:21.748317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.748184    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:32:21.748317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.748192    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:32:21.748317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.748276    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:32:21.750455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.750436    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"]
Nov 02 22:32:21.789857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.789832    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69clh\" (UniqueName: \"kubernetes.io/projected/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-kube-api-access-69clh\") pod \"maas-api-revoke-sub-b39e1aca-jvd6w\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.789988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.789877    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b39e1aca-jvd6w\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.789988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.789913    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b39e1aca-jvd6w\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.790522 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.790500    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b39e1aca-jvd6w\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.792190 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.792171    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b39e1aca-jvd6w\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.846531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.846502    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69clh\" (UniqueName: \"kubernetes.io/projected/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-kube-api-access-69clh\") pod \"maas-api-revoke-sub-b39e1aca-jvd6w\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:21.956207 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:21.956150    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:22.048144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:22.048102    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"]
Nov 02 22:32:22.051778 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:22.051754    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d37bd0-p5d8t"]
Nov 02 22:32:22.319856 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:32:22.319818    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6409ba_dd53_40fa_8840_8b4bb6c8c2eb.slice/crio-513bfc4d6f2475ee99d6059e1ffde495087086962227b527d316308a01070bbe WatchSource:0}: Error finding container 513bfc4d6f2475ee99d6059e1ffde495087086962227b527d316308a01070bbe: Status 404 returned error can't find the container with id 513bfc4d6f2475ee99d6059e1ffde495087086962227b527d316308a01070bbe
Nov 02 22:32:22.325371 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:22.325345    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"]
Nov 02 22:32:22.845393 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:22.845360    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w" event={"ID":"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb","Type":"ContainerStarted","Data":"d047a7ab0ec672fc5c76dd5cf3a1096002d81e0f707130c5cd3e5f531aa5bbdb"}
Nov 02 22:32:22.845742 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:22.845411    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w" event={"ID":"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb","Type":"ContainerStarted","Data":"513bfc4d6f2475ee99d6059e1ffde495087086962227b527d316308a01070bbe"}
Nov 02 22:32:22.912955 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:22.912924    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f71b322-9fb8-4f6c-8662-48c5fb7ee383" path="/var/lib/kubelet/pods/7f71b322-9fb8-4f6c-8662-48c5fb7ee383/volumes"
Nov 02 22:32:22.960377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:22.960333    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w" podStartSLOduration=1.9603170859999999 podStartE2EDuration="1.960317086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:22.958978336 +0000 UTC m=+5030.222514653" watchObservedRunningTime="2026-11-02 22:32:22.960317086 +0000 UTC m=+5030.223853402"
Nov 02 22:32:23.852594 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:23.852558    4528 generic.go:356] "Generic (PLEG): container finished" podID="bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb" containerID="d047a7ab0ec672fc5c76dd5cf3a1096002d81e0f707130c5cd3e5f531aa5bbdb" exitCode=0
Nov 02 22:32:23.852975 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:23.852646    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w" event={"ID":"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb","Type":"ContainerDied","Data":"d047a7ab0ec672fc5c76dd5cf3a1096002d81e0f707130c5cd3e5f531aa5bbdb"}
Nov 02 22:32:24.986556 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:24.986533    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:25.018829 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.018801    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-kube-api-access-69clh\" (UniqueName: \"kubernetes.io/projected/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-kube-api-access-69clh\") pod \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") "
Nov 02 22:32:25.018950 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.018859    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-xks-ca\") pod \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") "
Nov 02 22:32:25.018950 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.018898    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-openshift-ca\") pod \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\" (UID: \"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb\") "
Nov 02 22:32:25.019298 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.019271    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-openshift-ca" pod "bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb" (UID: "bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:32:25.020917 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.020880    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-xks-ca" pod "bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb" (UID: "bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:32:25.021002 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.020886    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-kube-api-access-69clh" pod "bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb" (UID: "bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb"). InnerVolumeSpecName "kube-api-access-69clh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:32:25.120121 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.120032    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69clh\" (UniqueName: \"kubernetes.io/projected/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-kube-api-access-69clh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:25.120121 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.120059    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:25.120121 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.120069    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:25.863756 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.863715    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w" event={"ID":"bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb","Type":"ContainerDied","Data":"513bfc4d6f2475ee99d6059e1ffde495087086962227b527d316308a01070bbe"}
Nov 02 22:32:25.863756 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.863763    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="513bfc4d6f2475ee99d6059e1ffde495087086962227b527d316308a01070bbe"
Nov 02 22:32:25.863979 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:25.863723    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"
Nov 02 22:32:32.323788 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.323757    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"]
Nov 02 22:32:32.324209 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.324159    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb" containerName="revoke-keys"
Nov 02 22:32:32.324209 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.324171    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb" containerName="revoke-keys"
Nov 02 22:32:32.324291 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.324249    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb" containerName="revoke-keys"
Nov 02 22:32:32.327498 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.327482    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.383885 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.383856    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"]
Nov 02 22:32:32.392313 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.392286    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26748248-06fb-4896-b3cc-15606b96016a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1407dd2-bg6ft\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.392454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.392391    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26748248-06fb-4896-b3cc-15606b96016a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1407dd2-bg6ft\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.392454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.392445    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4njj\" (UniqueName: \"kubernetes.io/projected/26748248-06fb-4896-b3cc-15606b96016a-kube-api-access-t4njj\") pod \"maas-api-revoke-sub-d1407dd2-bg6ft\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.431195 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.431163    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:32:32.431195 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.431185    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:32:32.431357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.431186    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:32:32.431357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.431167    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:32:32.493197 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.493169    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26748248-06fb-4896-b3cc-15606b96016a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1407dd2-bg6ft\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.493320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.493224    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4njj\" (UniqueName: \"kubernetes.io/projected/26748248-06fb-4896-b3cc-15606b96016a-kube-api-access-t4njj\") pod \"maas-api-revoke-sub-d1407dd2-bg6ft\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.493320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.493264    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26748248-06fb-4896-b3cc-15606b96016a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1407dd2-bg6ft\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.493902 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.493877    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26748248-06fb-4896-b3cc-15606b96016a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1407dd2-bg6ft\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.495797 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.495773    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26748248-06fb-4896-b3cc-15606b96016a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1407dd2-bg6ft\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.549945 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.549922    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4njj\" (UniqueName: \"kubernetes.io/projected/26748248-06fb-4896-b3cc-15606b96016a-kube-api-access-t4njj\") pod \"maas-api-revoke-sub-d1407dd2-bg6ft\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:32.639934 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:32.639865    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:33.008171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:33.008144    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"]
Nov 02 22:32:33.045590 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:33.045560    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"]
Nov 02 22:32:33.058208 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:33.058186    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e8cbb8-s5fbz"]
Nov 02 22:32:33.907595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:33.907561    4528 generic.go:356] "Generic (PLEG): container finished" podID="26748248-06fb-4896-b3cc-15606b96016a" containerID="dd266a46500abb4b0c71cbad59bef65de86e1ca407df28d032c356726c1d969c" exitCode=0
Nov 02 22:32:33.907943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:33.907642    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft" event={"ID":"26748248-06fb-4896-b3cc-15606b96016a","Type":"ContainerDied","Data":"dd266a46500abb4b0c71cbad59bef65de86e1ca407df28d032c356726c1d969c"}
Nov 02 22:32:33.907943 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:33.907686    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft" event={"ID":"26748248-06fb-4896-b3cc-15606b96016a","Type":"ContainerStarted","Data":"92ac3eb71f4ed775cd576c2451eb03f1e1ce349a8a27e3330f761e343d75c962"}
Nov 02 22:32:34.911995 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:34.911960    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab850fbc-874a-412f-9685-8e2e6487de13" path="/var/lib/kubelet/pods/ab850fbc-874a-412f-9685-8e2e6487de13/volumes"
Nov 02 22:32:35.048181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.048159    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:35.118722 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.118694    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26748248-06fb-4896-b3cc-15606b96016a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26748248-06fb-4896-b3cc-15606b96016a-maas-api-openshift-ca\") pod \"26748248-06fb-4896-b3cc-15606b96016a\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") "
Nov 02 22:32:35.118860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.118774    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26748248-06fb-4896-b3cc-15606b96016a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26748248-06fb-4896-b3cc-15606b96016a-maas-api-xks-ca\") pod \"26748248-06fb-4896-b3cc-15606b96016a\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") "
Nov 02 22:32:35.118860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.118817    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26748248-06fb-4896-b3cc-15606b96016a-kube-api-access-t4njj\" (UniqueName: \"kubernetes.io/projected/26748248-06fb-4896-b3cc-15606b96016a-kube-api-access-t4njj\") pod \"26748248-06fb-4896-b3cc-15606b96016a\" (UID: \"26748248-06fb-4896-b3cc-15606b96016a\") "
Nov 02 22:32:35.119021 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.118997    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26748248-06fb-4896-b3cc-15606b96016a-maas-api-openshift-ca" pod "26748248-06fb-4896-b3cc-15606b96016a" (UID: "26748248-06fb-4896-b3cc-15606b96016a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:32:35.120811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.120789    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26748248-06fb-4896-b3cc-15606b96016a-maas-api-xks-ca" pod "26748248-06fb-4896-b3cc-15606b96016a" (UID: "26748248-06fb-4896-b3cc-15606b96016a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:32:35.120909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.120789    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26748248-06fb-4896-b3cc-15606b96016a-kube-api-access-t4njj" pod "26748248-06fb-4896-b3cc-15606b96016a" (UID: "26748248-06fb-4896-b3cc-15606b96016a"). InnerVolumeSpecName "kube-api-access-t4njj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:32:35.220120 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.220036    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26748248-06fb-4896-b3cc-15606b96016a-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:35.220120 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.220064    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4njj\" (UniqueName: \"kubernetes.io/projected/26748248-06fb-4896-b3cc-15606b96016a-kube-api-access-t4njj\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:35.220120 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.220075    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26748248-06fb-4896-b3cc-15606b96016a-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:35.918789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.918754    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft" event={"ID":"26748248-06fb-4896-b3cc-15606b96016a","Type":"ContainerDied","Data":"92ac3eb71f4ed775cd576c2451eb03f1e1ce349a8a27e3330f761e343d75c962"}
Nov 02 22:32:35.918789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.918783    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"
Nov 02 22:32:35.918789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:35.918796    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92ac3eb71f4ed775cd576c2451eb03f1e1ce349a8a27e3330f761e343d75c962"
Nov 02 22:32:36.791927 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:36.791898    4528 scope.go:122] "RemoveContainer" containerID="8c6ae959acbb24a3a65d09dc1c070d707433117f6f6bf5bac3f17afaa453c25e"
Nov 02 22:32:36.800687 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:36.800669    4528 scope.go:122] "RemoveContainer" containerID="e102eab0f02795782cedc00c48ec5878c9192bc403a4962aaad23b33ed1ac547"
Nov 02 22:32:36.808452 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:36.808434    4528 scope.go:122] "RemoveContainer" containerID="e56a4044ca3670c99876143bb5bbd2201aa887e86e9cc0161c268f01b0f29cd9"
Nov 02 22:32:36.818482 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:36.818464    4528 scope.go:122] "RemoveContainer" containerID="cb3067305244d99ccee7feb3009aa3d52a80a7e07f139a0ce9e1679310608416"
Nov 02 22:32:36.825767 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:36.825751    4528 scope.go:122] "RemoveContainer" containerID="730ed30632ea122c70a95d118f92f17ccb04686b31a690e15b6c98d87743e81f"
Nov 02 22:32:37.286728 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:37.286700    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:32:43.928819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:43.928787    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"]
Nov 02 22:32:43.929236 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:43.929219    4528 cpu_manager.go:414] "containerMap: removing container" podUID="26748248-06fb-4896-b3cc-15606b96016a" containerName="revoke-keys"
Nov 02 22:32:43.929236 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:43.929232    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26748248-06fb-4896-b3cc-15606b96016a" containerName="revoke-keys"
Nov 02 22:32:43.929322 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:43.929306    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="26748248-06fb-4896-b3cc-15606b96016a" containerName="revoke-keys"
Nov 02 22:32:43.934417 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:43.934399    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:43.988550 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:43.988516    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"]
Nov 02 22:32:44.036096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.036062    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:32:44.036096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.036067    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:32:44.036096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.036090    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:32:44.037421 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.037406    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:32:44.099202 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.099175    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4e8959-l4gc5\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.099376 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.099310    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvnxn\" (UniqueName: \"kubernetes.io/projected/031695c6-2787-46f8-9f0f-05c289f7f583-kube-api-access-zvnxn\") pod \"maas-api-revoke-sub-bc4e8959-l4gc5\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.099423 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.099393    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4e8959-l4gc5\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.112358 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:32:44.112321    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:32:44.200838 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.200776    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4e8959-l4gc5\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.200960 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.200844    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvnxn\" (UniqueName: \"kubernetes.io/projected/031695c6-2787-46f8-9f0f-05c289f7f583-kube-api-access-zvnxn\") pod \"maas-api-revoke-sub-bc4e8959-l4gc5\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.200960 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.200896    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4e8959-l4gc5\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.201515 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.201496    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4e8959-l4gc5\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.203028 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.203009    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4e8959-l4gc5\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.258079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.258050    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvnxn\" (UniqueName: \"kubernetes.io/projected/031695c6-2787-46f8-9f0f-05c289f7f583-kube-api-access-zvnxn\") pod \"maas-api-revoke-sub-bc4e8959-l4gc5\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.547073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.547043    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:44.910690 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:32:44.910624    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031695c6_2787_46f8_9f0f_05c289f7f583.slice/crio-1b83d12e33a9bc712c8730d57f3308465de71f03870d74786e4e072217867a3b WatchSource:0}: Error finding container 1b83d12e33a9bc712c8730d57f3308465de71f03870d74786e4e072217867a3b: Status 404 returned error can't find the container with id 1b83d12e33a9bc712c8730d57f3308465de71f03870d74786e4e072217867a3b
Nov 02 22:32:44.913976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.913840    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"]
Nov 02 22:32:44.966972 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:44.966945    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5" event={"ID":"031695c6-2787-46f8-9f0f-05c289f7f583","Type":"ContainerStarted","Data":"1b83d12e33a9bc712c8730d57f3308465de71f03870d74786e4e072217867a3b"}
Nov 02 22:32:45.976535 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:45.976444    4528 generic.go:356] "Generic (PLEG): container finished" podID="031695c6-2787-46f8-9f0f-05c289f7f583" containerID="8318244b8c548c6ff353a9d21aeac562fdf759cb8e91b1849aa1d833ee695282" exitCode=0
Nov 02 22:32:45.976978 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:45.976530    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5" event={"ID":"031695c6-2787-46f8-9f0f-05c289f7f583","Type":"ContainerDied","Data":"8318244b8c548c6ff353a9d21aeac562fdf759cb8e91b1849aa1d833ee695282"}
Nov 02 22:32:47.137232 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.137204    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:47.225412 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.225382    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-openshift-ca\") pod \"031695c6-2787-46f8-9f0f-05c289f7f583\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") "
Nov 02 22:32:47.225600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.225499    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-xks-ca\") pod \"031695c6-2787-46f8-9f0f-05c289f7f583\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") "
Nov 02 22:32:47.225600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.225570    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031695c6-2787-46f8-9f0f-05c289f7f583-kube-api-access-zvnxn\" (UniqueName: \"kubernetes.io/projected/031695c6-2787-46f8-9f0f-05c289f7f583-kube-api-access-zvnxn\") pod \"031695c6-2787-46f8-9f0f-05c289f7f583\" (UID: \"031695c6-2787-46f8-9f0f-05c289f7f583\") "
Nov 02 22:32:47.225803 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.225774    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-openshift-ca" pod "031695c6-2787-46f8-9f0f-05c289f7f583" (UID: "031695c6-2787-46f8-9f0f-05c289f7f583"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:32:47.227490 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.227462    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-xks-ca" pod "031695c6-2787-46f8-9f0f-05c289f7f583" (UID: "031695c6-2787-46f8-9f0f-05c289f7f583"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:32:47.227623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.227603    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031695c6-2787-46f8-9f0f-05c289f7f583-kube-api-access-zvnxn" pod "031695c6-2787-46f8-9f0f-05c289f7f583" (UID: "031695c6-2787-46f8-9f0f-05c289f7f583"). InnerVolumeSpecName "kube-api-access-zvnxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:32:47.327131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.326818    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:47.327131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.326851    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvnxn\" (UniqueName: \"kubernetes.io/projected/031695c6-2787-46f8-9f0f-05c289f7f583-kube-api-access-zvnxn\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:47.327131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.326864    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031695c6-2787-46f8-9f0f-05c289f7f583-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:47.988438 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.988403    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5" event={"ID":"031695c6-2787-46f8-9f0f-05c289f7f583","Type":"ContainerDied","Data":"1b83d12e33a9bc712c8730d57f3308465de71f03870d74786e4e072217867a3b"}
Nov 02 22:32:47.988616 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.988453    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b83d12e33a9bc712c8730d57f3308465de71f03870d74786e4e072217867a3b"
Nov 02 22:32:47.988616 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:47.988411    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"
Nov 02 22:32:54.827791 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.827753    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"]
Nov 02 22:32:54.828389 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.828368    4528 cpu_manager.go:414] "containerMap: removing container" podUID="031695c6-2787-46f8-9f0f-05c289f7f583" containerName="revoke-keys"
Nov 02 22:32:54.828477 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.828390    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031695c6-2787-46f8-9f0f-05c289f7f583" containerName="revoke-keys"
Nov 02 22:32:54.828536 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.828515    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="031695c6-2787-46f8-9f0f-05c289f7f583" containerName="revoke-keys"
Nov 02 22:32:54.831579 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.831558    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:54.885460 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.885431    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:32:54.935577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.935544    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:32:54.935577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.935544    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:32:54.935795 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.935698    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\""
Nov 02 22:32:54.938005 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:54.937984    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"]
Nov 02 22:32:55.001273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.001240    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a371ee65-5g8h6\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.001439 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.001282    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqrr\" (UniqueName: \"kubernetes.io/projected/6cc573ba-957e-4d34-beca-4a1bea38e962-kube-api-access-fsqrr\") pod \"maas-api-revoke-sub-a371ee65-5g8h6\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.001439 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.001367    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a371ee65-5g8h6\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.102636 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.102557    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a371ee65-5g8h6\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.102798 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.102646    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a371ee65-5g8h6\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.102798 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.102671    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqrr\" (UniqueName: \"kubernetes.io/projected/6cc573ba-957e-4d34-beca-4a1bea38e962-kube-api-access-fsqrr\") pod \"maas-api-revoke-sub-a371ee65-5g8h6\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.103307 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.103283    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a371ee65-5g8h6\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.104874 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.104855    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a371ee65-5g8h6\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.159092 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.159060    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqrr\" (UniqueName: \"kubernetes.io/projected/6cc573ba-957e-4d34-beca-4a1bea38e962-kube-api-access-fsqrr\") pod \"maas-api-revoke-sub-a371ee65-5g8h6\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.443868 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.443782    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:55.808827 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:32:55.808797    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc573ba_957e_4d34_beca_4a1bea38e962.slice/crio-e14939327dd02377fde6bfd753c937ea681ca814d7446f3f7e692e4d88b4f012 WatchSource:0}: Error finding container e14939327dd02377fde6bfd753c937ea681ca814d7446f3f7e692e4d88b4f012: Status 404 returned error can't find the container with id e14939327dd02377fde6bfd753c937ea681ca814d7446f3f7e692e4d88b4f012
Nov 02 22:32:55.811550 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:55.811525    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"]
Nov 02 22:32:56.044761 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:56.044717    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6" event={"ID":"6cc573ba-957e-4d34-beca-4a1bea38e962","Type":"ContainerStarted","Data":"e14939327dd02377fde6bfd753c937ea681ca814d7446f3f7e692e4d88b4f012"}
Nov 02 22:32:57.051794 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:57.051763    4528 generic.go:356] "Generic (PLEG): container finished" podID="6cc573ba-957e-4d34-beca-4a1bea38e962" containerID="1cfd1512c7e1685e36212dd525bc7889f3526c4fa45477e5fd2540bef2352a6f" exitCode=0
Nov 02 22:32:57.052174 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:57.051849    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6" event={"ID":"6cc573ba-957e-4d34-beca-4a1bea38e962","Type":"ContainerDied","Data":"1cfd1512c7e1685e36212dd525bc7889f3526c4fa45477e5fd2540bef2352a6f"}
Nov 02 22:32:58.187181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.187155    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:58.330151 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.330055    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc573ba-957e-4d34-beca-4a1bea38e962-kube-api-access-fsqrr\" (UniqueName: \"kubernetes.io/projected/6cc573ba-957e-4d34-beca-4a1bea38e962-kube-api-access-fsqrr\") pod \"6cc573ba-957e-4d34-beca-4a1bea38e962\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") "
Nov 02 22:32:58.330347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.330159    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-openshift-ca\") pod \"6cc573ba-957e-4d34-beca-4a1bea38e962\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") "
Nov 02 22:32:58.330347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.330226    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-xks-ca\") pod \"6cc573ba-957e-4d34-beca-4a1bea38e962\" (UID: \"6cc573ba-957e-4d34-beca-4a1bea38e962\") "
Nov 02 22:32:58.330564 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.330534    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-openshift-ca" pod "6cc573ba-957e-4d34-beca-4a1bea38e962" (UID: "6cc573ba-957e-4d34-beca-4a1bea38e962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:32:58.332147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.332103    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-xks-ca" pod "6cc573ba-957e-4d34-beca-4a1bea38e962" (UID: "6cc573ba-957e-4d34-beca-4a1bea38e962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:32:58.332265 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.332238    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc573ba-957e-4d34-beca-4a1bea38e962-kube-api-access-fsqrr" pod "6cc573ba-957e-4d34-beca-4a1bea38e962" (UID: "6cc573ba-957e-4d34-beca-4a1bea38e962"). InnerVolumeSpecName "kube-api-access-fsqrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:32:58.431952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.431922    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsqrr\" (UniqueName: \"kubernetes.io/projected/6cc573ba-957e-4d34-beca-4a1bea38e962-kube-api-access-fsqrr\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:58.431952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.431949    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:58.432179 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:58.431960    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc573ba-957e-4d34-beca-4a1bea38e962-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:32:59.063704 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:59.063672    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"
Nov 02 22:32:59.063876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:59.063716    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6" event={"ID":"6cc573ba-957e-4d34-beca-4a1bea38e962","Type":"ContainerDied","Data":"e14939327dd02377fde6bfd753c937ea681ca814d7446f3f7e692e4d88b4f012"}
Nov 02 22:32:59.063876 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:59.063765    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e14939327dd02377fde6bfd753c937ea681ca814d7446f3f7e692e4d88b4f012"
Nov 02 22:32:59.678223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:32:59.678191    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:33:14.111562 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:33:14.111525    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:33:20.989539 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:20.989507    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:33:33.194851 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:33.194772    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"]
Nov 02 22:33:33.309321 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:33.309287    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:33:36.867184 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:36.867141    4528 scope.go:122] "RemoveContainer" containerID="c3797d95cd9fee605d47a2b8c282cdac36fb717d6f876fd36d28bb85a9d8dc86"
Nov 02 22:33:36.875833 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:36.875811    4528 scope.go:122] "RemoveContainer" containerID="f4e604a1f09fb5d9a3ad74b7c8f3484fc610665866d0246b71f25fd48ac2071a"
Nov 02 22:33:36.883296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:36.883275    4528 scope.go:122] "RemoveContainer" containerID="494a51490c8727a077a51c31536e3068f45f292e755c8c444d955caaa19357a1"
Nov 02 22:33:36.890973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:36.890954    4528 scope.go:122] "RemoveContainer" containerID="571fd54198e2e9420029c7103d5c1d20e69a4b0ef3353e5900b1cfa856e5cc31"
Nov 02 22:33:38.195802 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:38.195746    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f" podUID="389df98a-aff3-4a56-ab4a-998ef60e2bae" containerName="payload-pre-processing" containerID="cri-o://b3eee50480adb329d0b3aecaa3b9a36e88a7a9838971adefda08ea161b4bf8f9" gracePeriod=25
Nov 02 22:33:38.282841 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:38.282807    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f" podUID="389df98a-aff3-4a56-ab4a-998ef60e2bae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:33:44.111897 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:33:44.111855    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:33:48.282681 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:48.282649    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f" podUID="389df98a-aff3-4a56-ab4a-998ef60e2bae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:33:48.957300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:48.957268    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:33:53.174428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:53.174392    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"]
Nov 02 22:33:53.280521 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:53.280482    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"]
Nov 02 22:33:58.175172 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.175097    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl" podUID="96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" containerName="payload-processing" containerID="cri-o://9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e" gracePeriod=25
Nov 02 22:33:58.280982 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.280931    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw" podUID="53a2a5be-4801-4fc9-bb1f-5804be821e95" containerName="payload-pre-processing" containerID="cri-o://1d9ea3ae6e26992e090dff65bcef2030dd8df7d62059d474aa3a287cad2e5cf3" gracePeriod=25
Nov 02 22:33:58.281350 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:33:58.281321    4528 logging.go:55] [core] [Channel #21732 SubChannel #21733]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.238:50052", ServerName: "10.133.0.238:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.238:50052: connect: connection refused"
Nov 02 22:33:58.370785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.370755    4528 generic.go:356] "Generic (PLEG): container finished" podID="389df98a-aff3-4a56-ab4a-998ef60e2bae" containerID="b3eee50480adb329d0b3aecaa3b9a36e88a7a9838971adefda08ea161b4bf8f9" exitCode=0
Nov 02 22:33:58.370973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.370897    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f" event={"ID":"389df98a-aff3-4a56-ab4a-998ef60e2bae","Type":"ContainerDied","Data":"b3eee50480adb329d0b3aecaa3b9a36e88a7a9838971adefda08ea161b4bf8f9"}
Nov 02 22:33:58.433655 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.433601    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 22:33:58.453703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.453679    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389df98a-aff3-4a56-ab4a-998ef60e2bae-kube-api-access-kzzzw\" (UniqueName: \"kubernetes.io/projected/389df98a-aff3-4a56-ab4a-998ef60e2bae-kube-api-access-kzzzw\") pod \"389df98a-aff3-4a56-ab4a-998ef60e2bae\" (UID: \"389df98a-aff3-4a56-ab4a-998ef60e2bae\") "
Nov 02 22:33:58.453820 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.453721    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389df98a-aff3-4a56-ab4a-998ef60e2bae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389df98a-aff3-4a56-ab4a-998ef60e2bae-plugins-config-volume\") pod \"389df98a-aff3-4a56-ab4a-998ef60e2bae\" (UID: \"389df98a-aff3-4a56-ab4a-998ef60e2bae\") "
Nov 02 22:33:58.454144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.454100    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389df98a-aff3-4a56-ab4a-998ef60e2bae-plugins-config-volume" pod "389df98a-aff3-4a56-ab4a-998ef60e2bae" (UID: "389df98a-aff3-4a56-ab4a-998ef60e2bae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:33:58.455724 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.455706    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389df98a-aff3-4a56-ab4a-998ef60e2bae-kube-api-access-kzzzw" pod "389df98a-aff3-4a56-ab4a-998ef60e2bae" (UID: "389df98a-aff3-4a56-ab4a-998ef60e2bae"). InnerVolumeSpecName "kube-api-access-kzzzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:33:58.554769 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.554743    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzzzw\" (UniqueName: \"kubernetes.io/projected/389df98a-aff3-4a56-ab4a-998ef60e2bae-kube-api-access-kzzzw\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:58.554769 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:58.554767    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389df98a-aff3-4a56-ab4a-998ef60e2bae-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:33:59.282014 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:59.281978    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f" podUID="389df98a-aff3-4a56-ab4a-998ef60e2bae" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.238:50052\" within 1s: context deadline exceeded"
Nov 02 22:33:59.282423 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:33:59.282027    4528 logging.go:55] [core] [Channel #21732 SubChannel #21733]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.238:50052", ServerName: "10.133.0.238:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.238:50052: operation was canceled"
Nov 02 22:33:59.282423 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:59.282071    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 22:33:59.378236 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:59.378211    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"
Nov 02 22:33:59.378406 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:59.378211    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f" event={"ID":"389df98a-aff3-4a56-ab4a-998ef60e2bae","Type":"ContainerDied","Data":"3aaf43b07ef459785542b0a181f53642c3dcb9ec09516ad5f733a03dff95dc49"}
Nov 02 22:33:59.378406 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:59.378342    4528 scope.go:122] "RemoveContainer" containerID="b3eee50480adb329d0b3aecaa3b9a36e88a7a9838971adefda08ea161b4bf8f9"
Nov 02 22:33:59.543700 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:59.543614    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"]
Nov 02 22:33:59.594666 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:33:59.594633    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-97995b-7b8d7cb995-9bf2f"]
Nov 02 22:34:00.071958 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:00.071920    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl" podUID="96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:34:00.912200 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:00.912170    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389df98a-aff3-4a56-ab4a-998ef60e2bae" path="/var/lib/kubelet/pods/389df98a-aff3-4a56-ab4a-998ef60e2bae/volumes"
Nov 02 22:34:01.078235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:01.078200    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw" podUID="53a2a5be-4801-4fc9-bb1f-5804be821e95" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:34:05.901120 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:05.901081    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"]
Nov 02 22:34:05.901648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:05.901310    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy" containerID="cri-o://9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d" gracePeriod=30
Nov 02 22:34:06.046482 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:06.046449    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"]
Nov 02 22:34:06.070400 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:06.070375    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae51d07-8vtkh"]
Nov 02 22:34:06.911557 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:06.911514    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d907fbc4-2060-47ed-81f8-2067ee3f5a1c" path="/var/lib/kubelet/pods/d907fbc4-2060-47ed-81f8-2067ee3f5a1c/volumes"
Nov 02 22:34:07.045213 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:07.045183    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"]
Nov 02 22:34:07.095513 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:07.095483    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1baa6e4-d8pf4"]
Nov 02 22:34:08.912263 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:08.912231    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc" path="/var/lib/kubelet/pods/f7ba4b13-99a3-4d64-9a15-5907bd0ef7cc/volumes"
Nov 02 22:34:09.254404 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:09.254325    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:34:10.071762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:10.071724    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl" podUID="96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:34:11.078679 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.078642    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw" podUID="53a2a5be-4801-4fc9-bb1f-5804be821e95" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:34:11.146589 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.146563    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 22:34:11.262849 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.262828    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-data\") pod \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") "
Nov 02 22:34:11.262992 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.262869    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-token\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-token\") pod \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") "
Nov 02 22:34:11.262992 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.262951    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-kube-api-access-nt48b\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-kube-api-access-nt48b\") pod \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") "
Nov 02 22:34:11.262992 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.262984    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-socket\") pod \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") "
Nov 02 22:34:11.263182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263010    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-certs\") pod \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") "
Nov 02 22:34:11.263182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263038    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-envoy\") pod \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") "
Nov 02 22:34:11.263182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263067    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-podinfo\") pod \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") "
Nov 02 22:34:11.263182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263092    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-data" pod "1adfc106-9a49-44d2-b5c7-cca45733f6bd" (UID: "1adfc106-9a49-44d2-b5c7-cca45733f6bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:34:11.263182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263159    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istiod-ca-cert\") pod \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") "
Nov 02 22:34:11.263456 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263191    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-credential-socket\") pod \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\" (UID: \"1adfc106-9a49-44d2-b5c7-cca45733f6bd\") "
Nov 02 22:34:11.263456 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263287    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-socket" pod "1adfc106-9a49-44d2-b5c7-cca45733f6bd" (UID: "1adfc106-9a49-44d2-b5c7-cca45733f6bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:34:11.263456 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263341    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-certs" pod "1adfc106-9a49-44d2-b5c7-cca45733f6bd" (UID: "1adfc106-9a49-44d2-b5c7-cca45733f6bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:34:11.263624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263484    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-credential-socket" pod "1adfc106-9a49-44d2-b5c7-cca45733f6bd" (UID: "1adfc106-9a49-44d2-b5c7-cca45733f6bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:34:11.263624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263561    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:11.263624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263583    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:11.263624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263598    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:11.263624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263611    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:11.263897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.263652    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istiod-ca-cert" pod "1adfc106-9a49-44d2-b5c7-cca45733f6bd" (UID: "1adfc106-9a49-44d2-b5c7-cca45733f6bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:34:11.265156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.265133    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-token" pod "1adfc106-9a49-44d2-b5c7-cca45733f6bd" (UID: "1adfc106-9a49-44d2-b5c7-cca45733f6bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:34:11.265274 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.265176    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-podinfo" pod "1adfc106-9a49-44d2-b5c7-cca45733f6bd" (UID: "1adfc106-9a49-44d2-b5c7-cca45733f6bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:34:11.265529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.265509    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-envoy" pod "1adfc106-9a49-44d2-b5c7-cca45733f6bd" (UID: "1adfc106-9a49-44d2-b5c7-cca45733f6bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:34:11.265596 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.265528    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-kube-api-access-nt48b" pod "1adfc106-9a49-44d2-b5c7-cca45733f6bd" (UID: "1adfc106-9a49-44d2-b5c7-cca45733f6bd"). InnerVolumeSpecName "kube-api-access-nt48b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:34:11.316099 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.316065    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"]
Nov 02 22:34:11.364367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.364338    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:11.364367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.364367    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt48b\" (UniqueName: \"kubernetes.io/projected/1adfc106-9a49-44d2-b5c7-cca45733f6bd-kube-api-access-nt48b\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:11.364576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.364385    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:11.364576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.364398    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:11.364576 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.364410    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adfc106-9a49-44d2-b5c7-cca45733f6bd-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:11.442006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.441977    4528 generic.go:356] "Generic (PLEG): container finished" podID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerID="9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d" exitCode=0
Nov 02 22:34:11.442178 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.442025    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"
Nov 02 22:34:11.442178 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.442061    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" event={"ID":"1adfc106-9a49-44d2-b5c7-cca45733f6bd","Type":"ContainerDied","Data":"9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d"}
Nov 02 22:34:11.442178 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.442129    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc" event={"ID":"1adfc106-9a49-44d2-b5c7-cca45733f6bd","Type":"ContainerDied","Data":"40a57aa0927c54e6d051d1c413e5b528208660865019ba10601278ba7cf96fd1"}
Nov 02 22:34:11.442178 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.442156    4528 scope.go:122] "RemoveContainer" containerID="9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d"
Nov 02 22:34:11.451807 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.451789    4528 scope.go:122] "RemoveContainer" containerID="123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a"
Nov 02 22:34:11.459413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.459398    4528 scope.go:122] "RemoveContainer" containerID="9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d"
Nov 02 22:34:11.459650 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:34:11.459633    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d\": container with ID starting with 9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d not found: ID does not exist" containerID="9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d"
Nov 02 22:34:11.459691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.459659    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d"} err="failed to get container status \"9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d\": rpc error: code = NotFound desc = could not find container \"9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d\": container with ID starting with 9afbe7aef9d4b36d18443e42b91a14b2f47fee4b418519925d50a8dad8e6701d not found: ID does not exist"
Nov 02 22:34:11.459691 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.459674    4528 scope.go:122] "RemoveContainer" containerID="123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a"
Nov 02 22:34:11.459890 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:34:11.459874    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a\": container with ID starting with 123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a not found: ID does not exist" containerID="123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a"
Nov 02 22:34:11.459933 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.459896    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a"} err="failed to get container status \"123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a\": rpc error: code = NotFound desc = could not find container \"123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a\": container with ID starting with 123f1d2abc89c9dc3d3498cdcca8c2fc7e8a97495d213c6d0a2205d7cb8e0d6a not found: ID does not exist"
Nov 02 22:34:11.589909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.589836    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"]
Nov 02 22:34:11.593555 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:11.593530    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a1d751-openshift-default-5466c84b45-6vzkc"]
Nov 02 22:34:12.912679 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:12.912647    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" path="/var/lib/kubelet/pods/1adfc106-9a49-44d2-b5c7-cca45733f6bd/volumes"
Nov 02 22:34:14.112947 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:34:14.112906    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:34:18.428201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.428174    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 22:34:18.479634 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.479600    4528 generic.go:356] "Generic (PLEG): container finished" podID="96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" containerID="9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e" exitCode=0
Nov 02 22:34:18.479822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.479647    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"
Nov 02 22:34:18.479822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.479702    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl" event={"ID":"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6","Type":"ContainerDied","Data":"9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e"}
Nov 02 22:34:18.479822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.479752    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl" event={"ID":"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6","Type":"ContainerDied","Data":"4d301e51f47be4f8f520201f9bd97e2aa73428970283f706b9830ac9f3ed0655"}
Nov 02 22:34:18.479822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.479779    4528 scope.go:122] "RemoveContainer" containerID="9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e"
Nov 02 22:34:18.481600 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.481576    4528 generic.go:356] "Generic (PLEG): container finished" podID="53a2a5be-4801-4fc9-bb1f-5804be821e95" containerID="1d9ea3ae6e26992e090dff65bcef2030dd8df7d62059d474aa3a287cad2e5cf3" exitCode=0
Nov 02 22:34:18.481758 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.481670    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw" event={"ID":"53a2a5be-4801-4fc9-bb1f-5804be821e95","Type":"ContainerDied","Data":"1d9ea3ae6e26992e090dff65bcef2030dd8df7d62059d474aa3a287cad2e5cf3"}
Nov 02 22:34:18.489004 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.488975    4528 scope.go:122] "RemoveContainer" containerID="9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e"
Nov 02 22:34:18.489373 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:34:18.489346    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e\": container with ID starting with 9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e not found: ID does not exist" containerID="9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e"
Nov 02 22:34:18.489459 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.489379    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e"} err="failed to get container status \"9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e\": rpc error: code = NotFound desc = could not find container \"9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e\": container with ID starting with 9682e0a0986cce47f99b2f32dfaf1380b4e7788f995f8c52f8b2dc9e99e2946e not found: ID does not exist"
Nov 02 22:34:18.525166 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.525145    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 22:34:18.527957 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.527941    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-kube-api-access-bhmvx\" (UniqueName: \"kubernetes.io/projected/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-kube-api-access-bhmvx\") pod \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\" (UID: \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\") "
Nov 02 22:34:18.528038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.527977    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-plugins-config-volume\") pod \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\" (UID: \"96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6\") "
Nov 02 22:34:18.528344 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.528326    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-plugins-config-volume" pod "96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" (UID: "96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:34:18.529854 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.529834    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-kube-api-access-bhmvx" pod "96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" (UID: "96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6"). InnerVolumeSpecName "kube-api-access-bhmvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:34:18.629371 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.629340    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a2a5be-4801-4fc9-bb1f-5804be821e95-kube-api-access-mrcx5\" (UniqueName: \"kubernetes.io/projected/53a2a5be-4801-4fc9-bb1f-5804be821e95-kube-api-access-mrcx5\") pod \"53a2a5be-4801-4fc9-bb1f-5804be821e95\" (UID: \"53a2a5be-4801-4fc9-bb1f-5804be821e95\") "
Nov 02 22:34:18.629514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.629441    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a2a5be-4801-4fc9-bb1f-5804be821e95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a2a5be-4801-4fc9-bb1f-5804be821e95-plugins-config-volume\") pod \"53a2a5be-4801-4fc9-bb1f-5804be821e95\" (UID: \"53a2a5be-4801-4fc9-bb1f-5804be821e95\") "
Nov 02 22:34:18.629631 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.629607    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:18.629671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.629634    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhmvx\" (UniqueName: \"kubernetes.io/projected/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6-kube-api-access-bhmvx\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:18.629790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.629771    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a2a5be-4801-4fc9-bb1f-5804be821e95-plugins-config-volume" pod "53a2a5be-4801-4fc9-bb1f-5804be821e95" (UID: "53a2a5be-4801-4fc9-bb1f-5804be821e95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:34:18.631377 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.631357    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a2a5be-4801-4fc9-bb1f-5804be821e95-kube-api-access-mrcx5" pod "53a2a5be-4801-4fc9-bb1f-5804be821e95" (UID: "53a2a5be-4801-4fc9-bb1f-5804be821e95"). InnerVolumeSpecName "kube-api-access-mrcx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:34:18.730586 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.730512    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a2a5be-4801-4fc9-bb1f-5804be821e95-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:18.730586 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.730546    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrcx5\" (UniqueName: \"kubernetes.io/projected/53a2a5be-4801-4fc9-bb1f-5804be821e95-kube-api-access-mrcx5\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:18.899173 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.899130    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"]
Nov 02 22:34:18.901974 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.901950    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a1d751-7d8bfd776c-45sdl"]
Nov 02 22:34:18.911500 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:18.911478    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" path="/var/lib/kubelet/pods/96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6/volumes"
Nov 02 22:34:19.488361 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:19.488282    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"
Nov 02 22:34:19.488361 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:19.488316    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw" event={"ID":"53a2a5be-4801-4fc9-bb1f-5804be821e95","Type":"ContainerDied","Data":"907f926025659fb34b070d28dc0bd0bcd2e869e2c46956a14dd6f8c398dee1dc"}
Nov 02 22:34:19.488845 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:19.488378    4528 scope.go:122] "RemoveContainer" containerID="1d9ea3ae6e26992e090dff65bcef2030dd8df7d62059d474aa3a287cad2e5cf3"
Nov 02 22:34:19.651056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:19.651028    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"]
Nov 02 22:34:19.654916 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:19.654891    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a1d751-75b747db87-9v4gw"]
Nov 02 22:34:20.280745 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:20.280716    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"]
Nov 02 22:34:20.911896 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:20.911859    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a2a5be-4801-4fc9-bb1f-5804be821e95" path="/var/lib/kubelet/pods/53a2a5be-4801-4fc9-bb1f-5804be821e95/volumes"
Nov 02 22:34:22.047145 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:22.047091    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"]
Nov 02 22:34:22.098796 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:22.098769    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994d2ec6-s64j6"]
Nov 02 22:34:22.912089 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:22.912047    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a59620-b8fe-456f-afb8-eb95c0de11ac" path="/var/lib/kubelet/pods/34a59620-b8fe-456f-afb8-eb95c0de11ac/volumes"
Nov 02 22:34:24.047384 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:24.047352    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"]
Nov 02 22:34:24.097886 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:24.097842    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"]
Nov 02 22:34:24.099756 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:24.099738    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e235b9dd-pxlrl"]
Nov 02 22:34:24.101522 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:24.101497    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fc16c0-jv9jh"]
Nov 02 22:34:24.758960 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:24.758926    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:34:24.912762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:24.912731    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5429792d-7e77-4dff-85e5-60ad855937f8" path="/var/lib/kubelet/pods/5429792d-7e77-4dff-85e5-60ad855937f8/volumes"
Nov 02 22:34:24.913175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:24.913161    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4750f1-535c-43c2-a353-db31322f9df0" path="/var/lib/kubelet/pods/7e4750f1-535c-43c2-a353-db31322f9df0/volumes"
Nov 02 22:34:25.281075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:25.281036    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2" podUID="abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" containerName="payload-processing" containerID="cri-o://efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589" gracePeriod=25
Nov 02 22:34:26.334744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.334703    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main" containerID="cri-o://6457b5cd8f24b204a945849050c0cb8d674fe12a7c24850107cf38ab7a132bf7" gracePeriod=45
Nov 02 22:34:26.530073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.530018    4528 generic.go:356] "Generic (PLEG): container finished" podID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerID="6457b5cd8f24b204a945849050c0cb8d674fe12a7c24850107cf38ab7a132bf7" exitCode=0
Nov 02 22:34:26.530073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.530058    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" event={"ID":"a5f1c8cb-561b-400d-83a7-80686d1ba923","Type":"ContainerDied","Data":"6457b5cd8f24b204a945849050c0cb8d674fe12a7c24850107cf38ab7a132bf7"}
Nov 02 22:34:26.592038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.591977    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 22:34:26.702712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.702684    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-dshm\") pod \"a5f1c8cb-561b-400d-83a7-80686d1ba923\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") "
Nov 02 22:34:26.702914 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.702733    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f1c8cb-561b-400d-83a7-80686d1ba923-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c8cb-561b-400d-83a7-80686d1ba923-tls-certs\") pod \"a5f1c8cb-561b-400d-83a7-80686d1ba923\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") "
Nov 02 22:34:26.702914 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.702763    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-model-cache\") pod \"a5f1c8cb-561b-400d-83a7-80686d1ba923\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") "
Nov 02 22:34:26.702914 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.702884    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-home\") pod \"a5f1c8cb-561b-400d-83a7-80686d1ba923\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") "
Nov 02 22:34:26.703085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.702968    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f1c8cb-561b-400d-83a7-80686d1ba923-kube-api-access-7bblg\" (UniqueName: \"kubernetes.io/projected/a5f1c8cb-561b-400d-83a7-80686d1ba923-kube-api-access-7bblg\") pod \"a5f1c8cb-561b-400d-83a7-80686d1ba923\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") "
Nov 02 22:34:26.703085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.702985    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-model-cache" pod "a5f1c8cb-561b-400d-83a7-80686d1ba923" (UID: "a5f1c8cb-561b-400d-83a7-80686d1ba923"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:34:26.703085 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.703003    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-tmp-dir\") pod \"a5f1c8cb-561b-400d-83a7-80686d1ba923\" (UID: \"a5f1c8cb-561b-400d-83a7-80686d1ba923\") "
Nov 02 22:34:26.703270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.703089    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-home" pod "a5f1c8cb-561b-400d-83a7-80686d1ba923" (UID: "a5f1c8cb-561b-400d-83a7-80686d1ba923"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:34:26.703316 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.703267    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-tmp-dir" pod "a5f1c8cb-561b-400d-83a7-80686d1ba923" (UID: "a5f1c8cb-561b-400d-83a7-80686d1ba923"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:34:26.703431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.703417    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-model-cache\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:26.703468 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.703434    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-home\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:26.703468 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.703444    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-tmp-dir\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:26.704970 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.704942    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f1c8cb-561b-400d-83a7-80686d1ba923-tls-certs" pod "a5f1c8cb-561b-400d-83a7-80686d1ba923" (UID: "a5f1c8cb-561b-400d-83a7-80686d1ba923"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:34:26.705159 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.705142    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-dshm" pod "a5f1c8cb-561b-400d-83a7-80686d1ba923" (UID: "a5f1c8cb-561b-400d-83a7-80686d1ba923"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:34:26.705392 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.705371    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f1c8cb-561b-400d-83a7-80686d1ba923-kube-api-access-7bblg" pod "a5f1c8cb-561b-400d-83a7-80686d1ba923" (UID: "a5f1c8cb-561b-400d-83a7-80686d1ba923"). InnerVolumeSpecName "kube-api-access-7bblg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:34:26.804619 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.804585    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c8cb-561b-400d-83a7-80686d1ba923-tls-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:26.804619 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.804618    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bblg\" (UniqueName: \"kubernetes.io/projected/a5f1c8cb-561b-400d-83a7-80686d1ba923-kube-api-access-7bblg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:26.804619 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:26.804628    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c8cb-561b-400d-83a7-80686d1ba923-dshm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:27.538197 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:27.538155    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr" event={"ID":"a5f1c8cb-561b-400d-83a7-80686d1ba923","Type":"ContainerDied","Data":"14026c8c5991742eecd2189f054110095f18536a07e9c6771bf45fa01fd8e9cb"}
Nov 02 22:34:27.538197 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:27.538189    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"
Nov 02 22:34:27.538712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:27.538209    4528 scope.go:122] "RemoveContainer" containerID="6457b5cd8f24b204a945849050c0cb8d674fe12a7c24850107cf38ab7a132bf7"
Nov 02 22:34:27.705160 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:27.705125    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"]
Nov 02 22:34:27.754963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:27.754925    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/premium-w3-f51151-kserve-6cc686768-7vlmr"]
Nov 02 22:34:28.912593 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:28.912556    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" path="/var/lib/kubelet/pods/a5f1c8cb-561b-400d-83a7-80686d1ba923/volumes"
Nov 02 22:34:34.127345 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:34.127306    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2" podUID="abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:34:35.046736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:35.046703    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"]
Nov 02 22:34:35.096845 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:35.096817    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152b5df3-ghnj9"]
Nov 02 22:34:36.911198 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:36.911167    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a3eccb-6bf3-4188-8644-d67d36677817" path="/var/lib/kubelet/pods/24a3eccb-6bf3-4188-8644-d67d36677817/volumes"
Nov 02 22:34:39.641767 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:39.641728    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:34:44.111239 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:34:44.111205    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:34:44.127362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:44.127335    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2" podUID="abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:34:45.046171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.046139    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"]
Nov 02 22:34:45.096147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.096099    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46641806-qn7xj"]
Nov 02 22:34:45.519582 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.519560    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 22:34:45.626716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.626634    4528 generic.go:356] "Generic (PLEG): container finished" podID="abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" containerID="efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589" exitCode=0
Nov 02 22:34:45.626716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.626680    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"
Nov 02 22:34:45.626931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.626713    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2" event={"ID":"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0","Type":"ContainerDied","Data":"efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589"}
Nov 02 22:34:45.626931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.626762    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2" event={"ID":"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0","Type":"ContainerDied","Data":"c5d4c01db1d280ef50bc28baa83255da3fc2e4f018d654c1f894655fe9d2619a"}
Nov 02 22:34:45.626931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.626788    4528 scope.go:122] "RemoveContainer" containerID="efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589"
Nov 02 22:34:45.636176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.636156    4528 scope.go:122] "RemoveContainer" containerID="efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589"
Nov 02 22:34:45.636505 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:34:45.636482    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589\": container with ID starting with efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589 not found: ID does not exist" containerID="efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589"
Nov 02 22:34:45.636599 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.636517    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589"} err="failed to get container status \"efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589\": rpc error: code = NotFound desc = could not find container \"efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589\": container with ID starting with efccc71b1faf8fc5109c3fd18508fcf2d8b9e9d71ca9d3a1a9243362f3d69589 not found: ID does not exist"
Nov 02 22:34:45.672821 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.672791    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-plugins-config-volume\") pod \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\" (UID: \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\") "
Nov 02 22:34:45.672964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.672884    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-kube-api-access-xsksj\" (UniqueName: \"kubernetes.io/projected/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-kube-api-access-xsksj\") pod \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\" (UID: \"abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0\") "
Nov 02 22:34:45.673262 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.673234    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-plugins-config-volume" pod "abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" (UID: "abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:34:45.674879 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.674858    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-kube-api-access-xsksj" pod "abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" (UID: "abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0"). InnerVolumeSpecName "kube-api-access-xsksj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:34:45.774017 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.773985    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:45.774017 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:45.774012    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsksj\" (UniqueName: \"kubernetes.io/projected/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0-kube-api-access-xsksj\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:34:46.047299 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:46.047260    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"]
Nov 02 22:34:46.050504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:46.050480    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f51151-5475485995-h7mb2"]
Nov 02 22:34:46.912541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:46.912507    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321bace5-9613-4ad1-8b99-a2083bc2daf4" path="/var/lib/kubelet/pods/321bace5-9613-4ad1-8b99-a2083bc2daf4/volumes"
Nov 02 22:34:46.912932 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:46.912854    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" path="/var/lib/kubelet/pods/abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0/volumes"
Nov 02 22:34:48.495195 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495161    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"]
Nov 02 22:34:48.495748 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495726    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy"
Nov 02 22:34:48.495804 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495753    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy"
Nov 02 22:34:48.495804 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495771    4528 cpu_manager.go:414] "containerMap: removing container" podUID="53a2a5be-4801-4fc9-bb1f-5804be821e95" containerName="payload-pre-processing"
Nov 02 22:34:48.495804 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495779    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a2a5be-4801-4fc9-bb1f-5804be821e95" containerName="payload-pre-processing"
Nov 02 22:34:48.495804 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495798    4528 cpu_manager.go:414] "containerMap: removing container" podUID="389df98a-aff3-4a56-ab4a-998ef60e2bae" containerName="payload-pre-processing"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495806    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389df98a-aff3-4a56-ab4a-998ef60e2bae" containerName="payload-pre-processing"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495823    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6cc573ba-957e-4d34-beca-4a1bea38e962" containerName="revoke-keys"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495830    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc573ba-957e-4d34-beca-4a1bea38e962" containerName="revoke-keys"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495845    4528 cpu_manager.go:414] "containerMap: removing container" podUID="96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" containerName="payload-processing"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495852    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" containerName="payload-processing"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495859    4528 cpu_manager.go:414] "containerMap: removing container" podUID="abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" containerName="payload-processing"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495864    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" containerName="payload-processing"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495872    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495877    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main"
Nov 02 22:34:48.495951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495947    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a2a5be-4801-4fc9-bb1f-5804be821e95" containerName="payload-pre-processing"
Nov 02 22:34:48.496331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495958    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="389df98a-aff3-4a56-ab4a-998ef60e2bae" containerName="payload-pre-processing"
Nov 02 22:34:48.496331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495964    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc573ba-957e-4d34-beca-4a1bea38e962" containerName="revoke-keys"
Nov 02 22:34:48.496331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495970    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy"
Nov 02 22:34:48.496331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495975    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a7ebec-5acc-422b-b4e5-d6a0d6a4ace6" containerName="payload-processing"
Nov 02 22:34:48.496331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495981    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc82c18-b2a2-47cc-bfbe-4f48c80a5cb0" containerName="payload-processing"
Nov 02 22:34:48.496331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495987    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f1c8cb-561b-400d-83a7-80686d1ba923" containerName="main"
Nov 02 22:34:48.496331 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.495994    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy"
Nov 02 22:34:48.498987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.498972    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.550870 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.550844    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f51151-openshift-default-dockercfg-5nn46\""
Nov 02 22:34:48.596716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.596507    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.596716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.596556    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.596716 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.596584    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.596965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.596721    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4cf\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-kube-api-access-9l4cf\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.596965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.596787    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.596965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.596829    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.596965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.596897    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.596965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.596938    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6c1e94-5075-47bb-88c8-aa90ead6e809-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.597206 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.597019    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.603056 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.603034    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"]
Nov 02 22:34:48.698542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698512    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.698542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698547    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.698789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698566    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6c1e94-5075-47bb-88c8-aa90ead6e809-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.698789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698610    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.698789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698678    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.698972 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698785    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.698972 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698830    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.698972 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698865    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4cf\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-kube-api-access-9l4cf\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.698972 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698897    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.698972 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698923    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.699258 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.698972    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.699258 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.699057    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.699258 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.699215    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.699450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.699322    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6c1e94-5075-47bb-88c8-aa90ead6e809-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.700825 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.700801    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.701208 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.701185    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.755307 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.755252    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.755560 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.755539    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4cf\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-kube-api-access-9l4cf\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:48.811341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:48.811310    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:49.228624 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:34:49.228594    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6c1e94_5075_47bb_88c8_aa90ead6e809.slice/crio-fd3637c81621950e943a7ad0fb94d4e02b76f82a3598accb2e1b755d6d6a7012 WatchSource:0}: Error finding container fd3637c81621950e943a7ad0fb94d4e02b76f82a3598accb2e1b755d6d6a7012: Status 404 returned error can't find the container with id fd3637c81621950e943a7ad0fb94d4e02b76f82a3598accb2e1b755d6d6a7012
Nov 02 22:34:49.230712 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:49.230687    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"]
Nov 02 22:34:49.231059 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:49.231030    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:34:49.231144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:49.231089    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:34:49.231144 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:49.231133    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:34:49.651982 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:49.651944    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz" event={"ID":"be6c1e94-5075-47bb-88c8-aa90ead6e809","Type":"ContainerStarted","Data":"21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41"}
Nov 02 22:34:49.652427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:49.652003    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz" event={"ID":"be6c1e94-5075-47bb-88c8-aa90ead6e809","Type":"ContainerStarted","Data":"fd3637c81621950e943a7ad0fb94d4e02b76f82a3598accb2e1b755d6d6a7012"}
Nov 02 22:34:49.778531 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:49.778478    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz" podStartSLOduration=1.7784649780000001 podStartE2EDuration="1.778464978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:34:49.776523584 +0000 UTC m=+5177.040059902" watchObservedRunningTime="2026-11-02 22:34:49.778464978 +0000 UTC m=+5177.042001286"
Nov 02 22:34:49.822224 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:49.822185    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:34:50.824133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:50.822207    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Nov 02 22:34:50.824133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:50.822270    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz" podUID="be6c1e94-5075-47bb-88c8-aa90ead6e809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Nov 02 22:34:51.734980 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:51.734861    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"]
Nov 02 22:34:51.826135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:51.823466    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Nov 02 22:34:51.826135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:51.823524    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz" podUID="be6c1e94-5075-47bb-88c8-aa90ead6e809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Nov 02 22:34:52.665791 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:52.665753    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz" podUID="be6c1e94-5075-47bb-88c8-aa90ead6e809" containerName="istio-proxy" containerID="cri-o://21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41" gracePeriod=30
Nov 02 22:34:52.816130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:52.801214    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867cc8b678-sgzth_774878d1-3662-465a-8a44-3fa542eefd66/authorino/0.log"
Nov 02 22:34:55.381378 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:55.381342    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5tp5v_598f64cf-7fb5-4b62-a4ee-29802c83c4f4/limitador/0.log"
Nov 02 22:34:56.594033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:56.594004    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74879c7667-jgdxh_8ddc5d0a-c7a0-412a-ac84-c522dfcba78c/manager/0.log"
Nov 02 22:34:56.842765 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:56.842735    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"]
Nov 02 22:34:56.842965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:56.842946    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy" containerID="cri-o://6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371" gracePeriod=30
Nov 02 22:34:56.951419 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:56.951347    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9xjsn_4be9c715-baba-45d6-8cd9-19d679e10c2a/manager/0.log"
Nov 02 22:34:58.048465 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:34:58.048433    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rgnpj_cb32ed85-0c0b-4b06-95c0-8cf300e337f5/server/0.log"
Nov 02 22:35:00.612472 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.612448    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:35:00.706811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.706739    4528 generic.go:356] "Generic (PLEG): container finished" podID="be6c1e94-5075-47bb-88c8-aa90ead6e809" containerID="21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41" exitCode=0
Nov 02 22:35:00.706811 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.706782    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"
Nov 02 22:35:00.707038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.706822    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz" event={"ID":"be6c1e94-5075-47bb-88c8-aa90ead6e809","Type":"ContainerDied","Data":"21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41"}
Nov 02 22:35:00.707038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.706913    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz" event={"ID":"be6c1e94-5075-47bb-88c8-aa90ead6e809","Type":"ContainerDied","Data":"fd3637c81621950e943a7ad0fb94d4e02b76f82a3598accb2e1b755d6d6a7012"}
Nov 02 22:35:00.707038 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.706943    4528 scope.go:122] "RemoveContainer" containerID="21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41"
Nov 02 22:35:00.715156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715133    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-podinfo\") pod \"be6c1e94-5075-47bb-88c8-aa90ead6e809\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") "
Nov 02 22:35:00.715275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715165    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-socket\") pod \"be6c1e94-5075-47bb-88c8-aa90ead6e809\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") "
Nov 02 22:35:00.715275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715214    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6c1e94-5075-47bb-88c8-aa90ead6e809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6c1e94-5075-47bb-88c8-aa90ead6e809-istiod-ca-cert\") pod \"be6c1e94-5075-47bb-88c8-aa90ead6e809\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") "
Nov 02 22:35:00.715275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715236    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-token\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-token\") pod \"be6c1e94-5075-47bb-88c8-aa90ead6e809\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") "
Nov 02 22:35:00.715524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715286    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-envoy\") pod \"be6c1e94-5075-47bb-88c8-aa90ead6e809\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") "
Nov 02 22:35:00.715524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715311    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-data\") pod \"be6c1e94-5075-47bb-88c8-aa90ead6e809\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") "
Nov 02 22:35:00.715524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715326    4528 scope.go:122] "RemoveContainer" containerID="21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41"
Nov 02 22:35:00.715524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715352    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-kube-api-access-9l4cf\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-kube-api-access-9l4cf\") pod \"be6c1e94-5075-47bb-88c8-aa90ead6e809\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") "
Nov 02 22:35:00.715524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715412    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-certs\") pod \"be6c1e94-5075-47bb-88c8-aa90ead6e809\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") "
Nov 02 22:35:00.715524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715445    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-credential-socket\") pod \"be6c1e94-5075-47bb-88c8-aa90ead6e809\" (UID: \"be6c1e94-5075-47bb-88c8-aa90ead6e809\") "
Nov 02 22:35:00.715524 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715485    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-socket" pod "be6c1e94-5075-47bb-88c8-aa90ead6e809" (UID: "be6c1e94-5075-47bb-88c8-aa90ead6e809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:00.715905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715599    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-data" pod "be6c1e94-5075-47bb-88c8-aa90ead6e809" (UID: "be6c1e94-5075-47bb-88c8-aa90ead6e809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:00.715905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715712    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-certs" pod "be6c1e94-5075-47bb-88c8-aa90ead6e809" (UID: "be6c1e94-5075-47bb-88c8-aa90ead6e809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:00.715905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715642    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6c1e94-5075-47bb-88c8-aa90ead6e809-istiod-ca-cert" pod "be6c1e94-5075-47bb-88c8-aa90ead6e809" (UID: "be6c1e94-5075-47bb-88c8-aa90ead6e809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:35:00.715905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715786    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-credential-socket" pod "be6c1e94-5075-47bb-88c8-aa90ead6e809" (UID: "be6c1e94-5075-47bb-88c8-aa90ead6e809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:00.715905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715858    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6c1e94-5075-47bb-88c8-aa90ead6e809-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:00.715905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715879    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:00.715905 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715894    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:00.716235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715909    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:00.716235 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.715922    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:00.716235 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:35:00.716188    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41\": container with ID starting with 21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41 not found: ID does not exist" containerID="21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41"
Nov 02 22:35:00.716348 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.716226    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41"} err="failed to get container status \"21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41\": rpc error: code = NotFound desc = could not find container \"21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41\": container with ID starting with 21d36dae85d9115017ee4923276432ea0c48d3f8b4e28981814554ee9e9f5e41 not found: ID does not exist"
Nov 02 22:35:00.717414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.717394    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-podinfo" pod "be6c1e94-5075-47bb-88c8-aa90ead6e809" (UID: "be6c1e94-5075-47bb-88c8-aa90ead6e809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:35:00.717701 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.717680    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-envoy" pod "be6c1e94-5075-47bb-88c8-aa90ead6e809" (UID: "be6c1e94-5075-47bb-88c8-aa90ead6e809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:00.717927 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.717908    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-token" pod "be6c1e94-5075-47bb-88c8-aa90ead6e809" (UID: "be6c1e94-5075-47bb-88c8-aa90ead6e809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:35:00.717971 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.717931    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-kube-api-access-9l4cf" pod "be6c1e94-5075-47bb-88c8-aa90ead6e809" (UID: "be6c1e94-5075-47bb-88c8-aa90ead6e809"). InnerVolumeSpecName "kube-api-access-9l4cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:35:00.816313 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.816283    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4cf\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-kube-api-access-9l4cf\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:00.816313 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.816309    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:00.816468 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.816324    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:00.816468 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:00.816336    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6c1e94-5075-47bb-88c8-aa90ead6e809-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:01.050425 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:01.050397    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"]
Nov 02 22:35:01.053048 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:01.053025    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a9f62f-2ttml"]
Nov 02 22:35:01.119496 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:01.119469    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"]
Nov 02 22:35:01.123897 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:01.123875    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-8d2sz"]
Nov 02 22:35:02.091611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.091587    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:35:02.126169 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126095    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-credential-socket\") pod \"99c92629-8829-478d-8f30-a732283d238c\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") "
Nov 02 22:35:02.126169 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126154    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-kube-api-access-w9s75\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-kube-api-access-w9s75\") pod \"99c92629-8829-478d-8f30-a732283d238c\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") "
Nov 02 22:35:02.126368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126188    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-envoy\") pod \"99c92629-8829-478d-8f30-a732283d238c\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") "
Nov 02 22:35:02.126368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126216    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-data\") pod \"99c92629-8829-478d-8f30-a732283d238c\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") "
Nov 02 22:35:02.126368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126250    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99c92629-8829-478d-8f30-a732283d238c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c92629-8829-478d-8f30-a732283d238c-istio-podinfo\") pod \"99c92629-8829-478d-8f30-a732283d238c\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") "
Nov 02 22:35:02.126368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126278    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-certs\") pod \"99c92629-8829-478d-8f30-a732283d238c\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") "
Nov 02 22:35:02.126368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126336    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-credential-socket" pod "99c92629-8829-478d-8f30-a732283d238c" (UID: "99c92629-8829-478d-8f30-a732283d238c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:02.126368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126365    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-istio-token\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-istio-token\") pod \"99c92629-8829-478d-8f30-a732283d238c\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") "
Nov 02 22:35:02.126693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126394    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-socket\") pod \"99c92629-8829-478d-8f30-a732283d238c\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") "
Nov 02 22:35:02.126693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126431    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c92629-8829-478d-8f30-a732283d238c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c92629-8829-478d-8f30-a732283d238c-istiod-ca-cert\") pod \"99c92629-8829-478d-8f30-a732283d238c\" (UID: \"99c92629-8829-478d-8f30-a732283d238c\") "
Nov 02 22:35:02.126693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126492    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-data" pod "99c92629-8829-478d-8f30-a732283d238c" (UID: "99c92629-8829-478d-8f30-a732283d238c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:02.126693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126581    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-certs" pod "99c92629-8829-478d-8f30-a732283d238c" (UID: "99c92629-8829-478d-8f30-a732283d238c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:02.126693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126645    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-socket" pod "99c92629-8829-478d-8f30-a732283d238c" (UID: "99c92629-8829-478d-8f30-a732283d238c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:02.126969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126797    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:02.126969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126821    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:02.126969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126835    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:02.126969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126848    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:02.126969 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.126882    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c92629-8829-478d-8f30-a732283d238c-istiod-ca-cert" pod "99c92629-8829-478d-8f30-a732283d238c" (UID: "99c92629-8829-478d-8f30-a732283d238c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:35:02.128408 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.128379    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99c92629-8829-478d-8f30-a732283d238c-istio-podinfo" pod "99c92629-8829-478d-8f30-a732283d238c" (UID: "99c92629-8829-478d-8f30-a732283d238c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:35:02.128523 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.128459    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-istio-token" pod "99c92629-8829-478d-8f30-a732283d238c" (UID: "99c92629-8829-478d-8f30-a732283d238c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:35:02.128523 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.128475    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-kube-api-access-w9s75" pod "99c92629-8829-478d-8f30-a732283d238c" (UID: "99c92629-8829-478d-8f30-a732283d238c"). InnerVolumeSpecName "kube-api-access-w9s75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:35:02.128596 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.128577    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-envoy" pod "99c92629-8829-478d-8f30-a732283d238c" (UID: "99c92629-8829-478d-8f30-a732283d238c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:35:02.227770 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.227741    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c92629-8829-478d-8f30-a732283d238c-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:02.227770 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.227764    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:02.227770 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.227774    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c92629-8829-478d-8f30-a732283d238c-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:02.227977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.227783    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9s75\" (UniqueName: \"kubernetes.io/projected/99c92629-8829-478d-8f30-a732283d238c-kube-api-access-w9s75\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:02.227977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.227791    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c92629-8829-478d-8f30-a732283d238c-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:02.718686 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.718652    4528 generic.go:356] "Generic (PLEG): container finished" podID="99c92629-8829-478d-8f30-a732283d238c" containerID="6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371" exitCode=0
Nov 02 22:35:02.718864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.718718    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"
Nov 02 22:35:02.718864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.718740    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" event={"ID":"99c92629-8829-478d-8f30-a732283d238c","Type":"ContainerDied","Data":"6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371"}
Nov 02 22:35:02.718864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.718794    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb" event={"ID":"99c92629-8829-478d-8f30-a732283d238c","Type":"ContainerDied","Data":"71b1703e0af41dfed65f1d0f7fe9d9bb4fe3bad216d10e6808e608de7a37ba20"}
Nov 02 22:35:02.718864 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.718821    4528 scope.go:122] "RemoveContainer" containerID="6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371"
Nov 02 22:35:02.728677 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.728659    4528 scope.go:122] "RemoveContainer" containerID="6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371"
Nov 02 22:35:02.728928 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:35:02.728912    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371\": container with ID starting with 6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371 not found: ID does not exist" containerID="6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371"
Nov 02 22:35:02.729012 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.728948    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371"} err="failed to get container status \"6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371\": rpc error: code = NotFound desc = could not find container \"6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371\": container with ID starting with 6ae880e6d13001259c2724137c005e8e0c5aac9950037bf403cbd052ff5e3371 not found: ID does not exist"
Nov 02 22:35:02.884202 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.884167    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"]
Nov 02 22:35:02.911964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.911932    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a24acc-dc6f-4ef9-b75c-332992234503" path="/var/lib/kubelet/pods/22a24acc-dc6f-4ef9-b75c-332992234503/volumes"
Nov 02 22:35:02.912373 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.912359    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6c1e94-5075-47bb-88c8-aa90ead6e809" path="/var/lib/kubelet/pods/be6c1e94-5075-47bb-88c8-aa90ead6e809/volumes"
Nov 02 22:35:02.934268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:02.934246    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z62zb"]
Nov 02 22:35:04.918212 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:04.918172    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c92629-8829-478d-8f30-a732283d238c" path="/var/lib/kubelet/pods/99c92629-8829-478d-8f30-a732283d238c/volumes"
Nov 02 22:35:10.761065 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.761028    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"]
Nov 02 22:35:10.761663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.761543    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy"
Nov 02 22:35:10.761663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.761563    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adfc106-9a49-44d2-b5c7-cca45733f6bd" containerName="istio-proxy"
Nov 02 22:35:10.761663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.761591    4528 cpu_manager.go:414] "containerMap: removing container" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy"
Nov 02 22:35:10.761663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.761600    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy"
Nov 02 22:35:10.761663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.761611    4528 cpu_manager.go:414] "containerMap: removing container" podUID="be6c1e94-5075-47bb-88c8-aa90ead6e809" containerName="istio-proxy"
Nov 02 22:35:10.761663 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.761619    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6c1e94-5075-47bb-88c8-aa90ead6e809" containerName="istio-proxy"
Nov 02 22:35:10.762007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.761709    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c92629-8829-478d-8f30-a732283d238c" containerName="istio-proxy"
Nov 02 22:35:10.762007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.761723    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6c1e94-5075-47bb-88c8-aa90ead6e809" containerName="istio-proxy"
Nov 02 22:35:10.765483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.765460    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:10.805424 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.805399    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnt7w\" (UniqueName: \"kubernetes.io/projected/a04200db-5a66-4295-ac97-b72620919cef-kube-api-access-rnt7w\") pod \"payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc\" (UID: \"a04200db-5a66-4295-ac97-b72620919cef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:10.805649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.805629    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04200db-5a66-4295-ac97-b72620919cef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc\" (UID: \"a04200db-5a66-4295-ac97-b72620919cef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:10.820209 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.820187    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d64a7c7d\""
Nov 02 22:35:10.870273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.870249    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d64a7c7d-dockercfg-2x9hn\""
Nov 02 22:35:10.871895 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.871874    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"]
Nov 02 22:35:10.907083 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.907051    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04200db-5a66-4295-ac97-b72620919cef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc\" (UID: \"a04200db-5a66-4295-ac97-b72620919cef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:10.907244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.907104    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnt7w\" (UniqueName: \"kubernetes.io/projected/a04200db-5a66-4295-ac97-b72620919cef-kube-api-access-rnt7w\") pod \"payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc\" (UID: \"a04200db-5a66-4295-ac97-b72620919cef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:10.907913 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.907896    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04200db-5a66-4295-ac97-b72620919cef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc\" (UID: \"a04200db-5a66-4295-ac97-b72620919cef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:10.963779 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:10.963754    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnt7w\" (UniqueName: \"kubernetes.io/projected/a04200db-5a66-4295-ac97-b72620919cef-kube-api-access-rnt7w\") pod \"payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc\" (UID: \"a04200db-5a66-4295-ac97-b72620919cef\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:11.078181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:11.078093    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:11.444607 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:35:11.444562    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04200db_5a66_4295_ac97_b72620919cef.slice/crio-5b1ca3f7819b3e723a995d6258098cc1cb5557c095d366951f470352dd183753 WatchSource:0}: Error finding container 5b1ca3f7819b3e723a995d6258098cc1cb5557c095d366951f470352dd183753: Status 404 returned error can't find the container with id 5b1ca3f7819b3e723a995d6258098cc1cb5557c095d366951f470352dd183753
Nov 02 22:35:11.445354 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:11.445316    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"]
Nov 02 22:35:11.764258 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:11.764228    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc" event={"ID":"a04200db-5a66-4295-ac97-b72620919cef","Type":"ContainerStarted","Data":"e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15"}
Nov 02 22:35:11.764648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:11.764282    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc" event={"ID":"a04200db-5a66-4295-ac97-b72620919cef","Type":"ContainerStarted","Data":"5b1ca3f7819b3e723a995d6258098cc1cb5557c095d366951f470352dd183753"}
Nov 02 22:35:11.764648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:11.764325    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:11.877033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:11.876987    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc" podStartSLOduration=1.876972711 podStartE2EDuration="1.876972711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:35:11.876353859 +0000 UTC m=+5199.139890177" watchObservedRunningTime="2026-11-02 22:35:11.876972711 +0000 UTC m=+5199.140509027"
Nov 02 22:35:14.113066 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:35:14.113032    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:35:21.880689 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:21.880651    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:35:22.772589 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:22.772561    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:36.947536 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:36.947505    4528 scope.go:122] "RemoveContainer" containerID="29d318222050759608512a6740aa89c98286aed87eb6cfefa4cae5e73f7b7633"
Nov 02 22:35:36.955555 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:36.955534    4528 scope.go:122] "RemoveContainer" containerID="643b1b48919b767c6515a64c6f9c18f659b9cb12a59cf982e38565c4edbac1e6"
Nov 02 22:35:36.962819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:36.962802    4528 scope.go:122] "RemoveContainer" containerID="31cd6554c578151697d288d472b59e3b512350a1927eb606dc81c169bc92b721"
Nov 02 22:35:36.970153 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:36.970136    4528 scope.go:122] "RemoveContainer" containerID="2c440de76c019a994a3135345c74ddad93a59e6e38025ae57d44db9bf1f481ff"
Nov 02 22:35:36.977387 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:36.977371    4528 scope.go:122] "RemoveContainer" containerID="fb6c6898b5ae63ea969f24898ddb74c1b0d4e0db1c957c1d027e2ef85cc9e662"
Nov 02 22:35:36.984329 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:36.984311    4528 scope.go:122] "RemoveContainer" containerID="51ee0317de288ecd04fd411f610161a7f2b401c102e8e7d9c21e41a69417496d"
Nov 02 22:35:43.470191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.470154    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"]
Nov 02 22:35:43.475262 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.475241    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.530436 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.530411    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"]
Nov 02 22:35:43.578428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.578397    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:35:43.578428 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.578415    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:35:43.578623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.578436    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:35:43.578623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.578472    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:35:43.587624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.587597    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrhl\" (UniqueName: \"kubernetes.io/projected/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-kube-api-access-9vrhl\") pod \"maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.587805 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.587630    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.587805 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.587769    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.688459 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.688433    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrhl\" (UniqueName: \"kubernetes.io/projected/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-kube-api-access-9vrhl\") pod \"maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.688601 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.688467    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.688601 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.688508    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.689127 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.689086    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.690835 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.690816    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.745251 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.745195    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrhl\" (UniqueName: \"kubernetes.io/projected/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-kube-api-access-9vrhl\") pod \"maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:43.786813 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:43.786792    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:44.112656 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:35:44.112624    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:35:44.150729 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:35:44.150699    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0374e3bb_618f_4deb_b20a_a3bd4152eb9d.slice/crio-e0312f3c86329c699e5e5a92e95c679938e6841e912f9e3fbf5af5a1e4fcafe3 WatchSource:0}: Error finding container e0312f3c86329c699e5e5a92e95c679938e6841e912f9e3fbf5af5a1e4fcafe3: Status 404 returned error can't find the container with id e0312f3c86329c699e5e5a92e95c679938e6841e912f9e3fbf5af5a1e4fcafe3
Nov 02 22:35:44.152817 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:44.152796    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"]
Nov 02 22:35:44.927319 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:44.927286    4528 generic.go:356] "Generic (PLEG): container finished" podID="0374e3bb-618f-4deb-b20a-a3bd4152eb9d" containerID="605ecd7843768a88cede78befcfce7b7aa996315b50ee10ed9422cb42d497489" exitCode=0
Nov 02 22:35:44.927759 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:44.927382    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt" event={"ID":"0374e3bb-618f-4deb-b20a-a3bd4152eb9d","Type":"ContainerDied","Data":"605ecd7843768a88cede78befcfce7b7aa996315b50ee10ed9422cb42d497489"}
Nov 02 22:35:44.927759 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:44.927449    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt" event={"ID":"0374e3bb-618f-4deb-b20a-a3bd4152eb9d","Type":"ContainerStarted","Data":"e0312f3c86329c699e5e5a92e95c679938e6841e912f9e3fbf5af5a1e4fcafe3"}
Nov 02 22:35:46.057533 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.057508    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:46.106285 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.106261    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-kube-api-access-9vrhl\" (UniqueName: \"kubernetes.io/projected/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-kube-api-access-9vrhl\") pod \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") "
Nov 02 22:35:46.106398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.106307    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-xks-ca\") pod \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") "
Nov 02 22:35:46.106398 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.106338    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-openshift-ca\") pod \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\" (UID: \"0374e3bb-618f-4deb-b20a-a3bd4152eb9d\") "
Nov 02 22:35:46.106733 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.106711    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-openshift-ca" pod "0374e3bb-618f-4deb-b20a-a3bd4152eb9d" (UID: "0374e3bb-618f-4deb-b20a-a3bd4152eb9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:35:46.108269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.108249    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-xks-ca" pod "0374e3bb-618f-4deb-b20a-a3bd4152eb9d" (UID: "0374e3bb-618f-4deb-b20a-a3bd4152eb9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:35:46.108369 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.108351    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-kube-api-access-9vrhl" pod "0374e3bb-618f-4deb-b20a-a3bd4152eb9d" (UID: "0374e3bb-618f-4deb-b20a-a3bd4152eb9d"). InnerVolumeSpecName "kube-api-access-9vrhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:35:46.206850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.206785    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vrhl\" (UniqueName: \"kubernetes.io/projected/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-kube-api-access-9vrhl\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:46.206850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.206811    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:46.206850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.206821    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0374e3bb-618f-4deb-b20a-a3bd4152eb9d-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:35:46.936953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.936913    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt" event={"ID":"0374e3bb-618f-4deb-b20a-a3bd4152eb9d","Type":"ContainerDied","Data":"e0312f3c86329c699e5e5a92e95c679938e6841e912f9e3fbf5af5a1e4fcafe3"}
Nov 02 22:35:46.936953 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.936927    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"
Nov 02 22:35:46.937157 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:46.936971    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0312f3c86329c699e5e5a92e95c679938e6841e912f9e3fbf5af5a1e4fcafe3"
Nov 02 22:35:54.669572 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:54.669538    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"]
Nov 02 22:35:58.870733 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:58.870696    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:35:59.196214 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:59.196133    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"]
Nov 02 22:35:59.246435 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:59.246402    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d64a7c7d-nxnbt"]
Nov 02 22:35:59.670097 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:59.670052    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc" podUID="a04200db-5a66-4295-ac97-b72620919cef" containerName="payload-pre-processing" containerID="cri-o://e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15" gracePeriod=25
Nov 02 22:35:59.926003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:59.925947    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:59.994510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:59.994469    4528 generic.go:356] "Generic (PLEG): container finished" podID="a04200db-5a66-4295-ac97-b72620919cef" containerID="e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15" exitCode=0
Nov 02 22:35:59.994671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:59.994569    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"
Nov 02 22:35:59.994671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:59.994561    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc" event={"ID":"a04200db-5a66-4295-ac97-b72620919cef","Type":"ContainerDied","Data":"e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15"}
Nov 02 22:35:59.994797 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:59.994768    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc" event={"ID":"a04200db-5a66-4295-ac97-b72620919cef","Type":"ContainerDied","Data":"5b1ca3f7819b3e723a995d6258098cc1cb5557c095d366951f470352dd183753"}
Nov 02 22:35:59.994857 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:35:59.994829    4528 scope.go:122] "RemoveContainer" containerID="e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15"
Nov 02 22:36:00.004073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.004054    4528 scope.go:122] "RemoveContainer" containerID="e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15"
Nov 02 22:36:00.004386 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:36:00.004367    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15\": container with ID starting with e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15 not found: ID does not exist" containerID="e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15"
Nov 02 22:36:00.004462 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.004394    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15"} err="failed to get container status \"e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15\": rpc error: code = NotFound desc = could not find container \"e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15\": container with ID starting with e3aa3004acb2c41045810da8653204453a42fbe5379cbf1a6fbaacfbb01fbc15 not found: ID does not exist"
Nov 02 22:36:00.021499 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.021479    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a04200db-5a66-4295-ac97-b72620919cef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04200db-5a66-4295-ac97-b72620919cef-plugins-config-volume\") pod \"a04200db-5a66-4295-ac97-b72620919cef\" (UID: \"a04200db-5a66-4295-ac97-b72620919cef\") "
Nov 02 22:36:00.021606 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.021570    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04200db-5a66-4295-ac97-b72620919cef-kube-api-access-rnt7w\" (UniqueName: \"kubernetes.io/projected/a04200db-5a66-4295-ac97-b72620919cef-kube-api-access-rnt7w\") pod \"a04200db-5a66-4295-ac97-b72620919cef\" (UID: \"a04200db-5a66-4295-ac97-b72620919cef\") "
Nov 02 22:36:00.021812 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.021792    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04200db-5a66-4295-ac97-b72620919cef-plugins-config-volume" pod "a04200db-5a66-4295-ac97-b72620919cef" (UID: "a04200db-5a66-4295-ac97-b72620919cef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:36:00.023455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.023426    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04200db-5a66-4295-ac97-b72620919cef-kube-api-access-rnt7w" pod "a04200db-5a66-4295-ac97-b72620919cef" (UID: "a04200db-5a66-4295-ac97-b72620919cef"). InnerVolumeSpecName "kube-api-access-rnt7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:36:00.122381 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.122353    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04200db-5a66-4295-ac97-b72620919cef-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:36:00.122381 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.122376    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnt7w\" (UniqueName: \"kubernetes.io/projected/a04200db-5a66-4295-ac97-b72620919cef-kube-api-access-rnt7w\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:36:00.416153 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.416103    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"]
Nov 02 22:36:00.466924 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.466893    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d64a7c7d-6bf78696c-4xftc"]
Nov 02 22:36:00.912140 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.912092    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0374e3bb-618f-4deb-b20a-a3bd4152eb9d" path="/var/lib/kubelet/pods/0374e3bb-618f-4deb-b20a-a3bd4152eb9d/volumes"
Nov 02 22:36:00.912481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:00.912469    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a04200db-5a66-4295-ac97-b72620919cef" path="/var/lib/kubelet/pods/a04200db-5a66-4295-ac97-b72620919cef/volumes"
Nov 02 22:36:14.111374 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:36:14.111334    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:36:25.044904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:25.044872    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"]
Nov 02 22:36:25.094736 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:25.094706    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7942a61c-wn2rg"]
Nov 02 22:36:26.910977 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:26.910941    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15617ad-4b5e-4693-9f9b-ee3ee8e674cc" path="/var/lib/kubelet/pods/d15617ad-4b5e-4693-9f9b-ee3ee8e674cc/volumes"
Nov 02 22:36:27.545976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.545940    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"]
Nov 02 22:36:27.546357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.546343    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0374e3bb-618f-4deb-b20a-a3bd4152eb9d" containerName="revoke-keys"
Nov 02 22:36:27.546413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.546359    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0374e3bb-618f-4deb-b20a-a3bd4152eb9d" containerName="revoke-keys"
Nov 02 22:36:27.546413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.546372    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a04200db-5a66-4295-ac97-b72620919cef" containerName="payload-pre-processing"
Nov 02 22:36:27.546413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.546378    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04200db-5a66-4295-ac97-b72620919cef" containerName="payload-pre-processing"
Nov 02 22:36:27.546515 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.546447    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0374e3bb-618f-4deb-b20a-a3bd4152eb9d" containerName="revoke-keys"
Nov 02 22:36:27.546515 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.546454    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04200db-5a66-4295-ac97-b72620919cef" containerName="payload-pre-processing"
Nov 02 22:36:27.549506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.549491    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.601570 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.601546    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4lw9\""
Nov 02 22:36:27.664724 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.664693    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.664881 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.664734    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.664881 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.664762    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.664881 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.664844    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.664999 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.664908    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.664999 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.664935    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvfb\" (UniqueName: \"kubernetes.io/projected/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-kube-api-access-njvfb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.665100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.665028    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.665100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.665055    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.665194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.665139    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.757601 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.757566    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"]
Nov 02 22:36:27.765597 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.765567    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.765744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.765606    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.765744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.765652    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.765744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.765687    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.765744 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.765727    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.765992 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.765756    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.765992 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.765782    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.765992 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.765817    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.765992 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.765843    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvfb\" (UniqueName: \"kubernetes.io/projected/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-kube-api-access-njvfb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.766240 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.766034    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.766240 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.766070    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.766362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.766252    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.766483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.766444    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.766483 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.766472    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.768160 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.768138    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.768269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.768211    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.821963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.821904    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.822060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.822045    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvfb\" (UniqueName: \"kubernetes.io/projected/00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd-kube-api-access-njvfb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-pwm9l\" (UID: \"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:27.862060 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:27.862034    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:28.281270 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:36:28.280986    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ba69b8_9f85_4cd0_bad9_b8bc4bd752fd.slice/crio-41619878a8d2c7dde1cd64960511b80a8f9829c8f5df0bc38ec2d9c6e320c5c5 WatchSource:0}: Error finding container 41619878a8d2c7dde1cd64960511b80a8f9829c8f5df0bc38ec2d9c6e320c5c5: Status 404 returned error can't find the container with id 41619878a8d2c7dde1cd64960511b80a8f9829c8f5df0bc38ec2d9c6e320c5c5
Nov 02 22:36:28.282563 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:28.282535    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"]
Nov 02 22:36:28.283297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:28.283269    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:36:28.283359 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:28.283328    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:36:28.283439 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:28.283359    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:36:29.131611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:29.131577    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" event={"ID":"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd","Type":"ContainerStarted","Data":"c30af0eb7927170b538ffcb0e755b5e3cb017b906c811c93e2a88f5dbd6c011c"}
Nov 02 22:36:29.131807 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:29.131634    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" event={"ID":"00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd","Type":"ContainerStarted","Data":"41619878a8d2c7dde1cd64960511b80a8f9829c8f5df0bc38ec2d9c6e320c5c5"}
Nov 02 22:36:29.347546 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:29.347489    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" podStartSLOduration=2.347472913 podStartE2EDuration="2.347472913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:29.297073559 +0000 UTC m=+5276.560609877" watchObservedRunningTime="2026-11-02 22:36:29.347472913 +0000 UTC m=+5276.611009231"
Nov 02 22:36:29.862609 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:29.862579    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:29.864269 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:29.864242    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pwm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused" start-of-body=
Nov 02 22:36:29.864404 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:29.864291    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" podUID="00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused"
Nov 02 22:36:30.863081 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:30.863048    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pwm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused" start-of-body=
Nov 02 22:36:30.863505 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:30.863097    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" podUID="00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused"
Nov 02 22:36:31.863364 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:31.863324    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pwm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused" start-of-body=
Nov 02 22:36:31.863752 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:31.863378    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" podUID="00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused"
Nov 02 22:36:32.863088 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:32.863061    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pwm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused" start-of-body=
Nov 02 22:36:32.863253 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:32.863135    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" podUID="00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused"
Nov 02 22:36:33.862593 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:33.862549    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pwm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused" start-of-body=
Nov 02 22:36:33.863072 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:33.862609    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" podUID="00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused"
Nov 02 22:36:34.862460 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:34.862425    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pwm9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused" start-of-body=
Nov 02 22:36:34.862673 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:34.862493    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" podUID="00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.121:15021/healthz/ready\": dial tcp 10.133.1.121:15021: connect: connection refused"
Nov 02 22:36:36.295931 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:36.295897    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:36.296438 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:36.296333    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:36.297244 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:36.297226    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l"
Nov 02 22:36:37.032188 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:37.032161    4528 scope.go:122] "RemoveContainer" containerID="2c694fba54f5d6e11a331cb3f31d85e4d77a705c864ed32274a6e268cfea915e"
Nov 02 22:36:37.040438 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:37.040419    4528 scope.go:122] "RemoveContainer" containerID="a6f359ba5bfcd71cb563895f735d56c642cca82dc51d1191262b434b052f237c"
Nov 02 22:36:37.045829 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:37.045804    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"]
Nov 02 22:36:37.047537 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:37.047518    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-379f854e-h7tms"]
Nov 02 22:36:38.911101 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:36:38.911068    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4a91b7-27ab-4795-a2c7-14816da0e206" path="/var/lib/kubelet/pods/ae4a91b7-27ab-4795-a2c7-14816da0e206/volumes"
Nov 02 22:36:44.113275 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:36:44.113240    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:37:14.111938 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:37:14.111899    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:37:16.861072 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:16.861037    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:37:25.046343 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:25.046313    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"]
Nov 02 22:37:25.096130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:25.096090    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b39e1aca-jvd6w"]
Nov 02 22:37:26.912305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:26.912269    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb" path="/var/lib/kubelet/pods/bd6409ba-dd53-40fa-8840-8b4bb6c8c2eb/volumes"
Nov 02 22:37:35.561949 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:35.561915    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:37:36.044370 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:36.044337    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"]
Nov 02 22:37:36.094290 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:36.094260    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1407dd2-bg6ft"]
Nov 02 22:37:36.911789 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:36.911757    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26748248-06fb-4896-b3cc-15606b96016a" path="/var/lib/kubelet/pods/26748248-06fb-4896-b3cc-15606b96016a/volumes"
Nov 02 22:37:37.065364 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:37.065331    4528 scope.go:122] "RemoveContainer" containerID="95f96cc1028913c9ace1ad951eee4090999ddaa95acf3e7fbff547c9dc0cbef1"
Nov 02 22:37:37.073236 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:37.073218    4528 scope.go:122] "RemoveContainer" containerID="31245f800c6e2577058a18af1713b4dfcc85c13f37ec1dfddc5615f077e709d8"
Nov 02 22:37:43.073209 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.073173    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"]
Nov 02 22:37:43.078099 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.078082    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.119540 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.119511    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0bbf59-4728-4a9a-882a-44eec7d799df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.119690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.119569    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.119690 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.119666    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.119785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.119717    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.119785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.119765    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.119873 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.119791    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.119873 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.119812    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62gmp\" (UniqueName: \"kubernetes.io/projected/2a0bbf59-4728-4a9a-882a-44eec7d799df-kube-api-access-62gmp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.119957 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.119870    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.119957 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.119901    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.144670 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.144645    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"]
Nov 02 22:37:43.148027 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.148010    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.198046 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.198020    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"]
Nov 02 22:37:43.220987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.220958    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.221209 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221182    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddfda63-2274-4793-bb2a-e99d52000804-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.221401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221244    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.221401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221270    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzqr\" (UniqueName: \"kubernetes.io/projected/5ddfda63-2274-4793-bb2a-e99d52000804-kube-api-access-2gzqr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.221401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221302    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.221401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221331    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.221401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221349    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.221401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221387    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddfda63-2274-4793-bb2a-e99d52000804-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.221776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221645    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.221839 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221780    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.221839 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221818    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.221946 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221849    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.221946 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221875    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.222062 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.221993    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62gmp\" (UniqueName: \"kubernetes.io/projected/2a0bbf59-4728-4a9a-882a-44eec7d799df-kube-api-access-62gmp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.222062 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.222037    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.222205 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.222063    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.222205 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.222089    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.222205 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.222129    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0bbf59-4728-4a9a-882a-44eec7d799df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.222205 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.222158    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.222414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.222200    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.222414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.222204    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddfda63-2274-4793-bb2a-e99d52000804-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.222546 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.222455    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.223238 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.223216    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0bbf59-4728-4a9a-882a-44eec7d799df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.224374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.224351    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.224588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.224566    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.244987 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.244964    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"]
Nov 02 22:37:43.278982 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.278954    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0bbf59-4728-4a9a-882a-44eec7d799df-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.279282 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.279260    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62gmp\" (UniqueName: \"kubernetes.io/projected/2a0bbf59-4728-4a9a-882a-44eec7d799df-kube-api-access-62gmp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gj9qp\" (UID: \"2a0bbf59-4728-4a9a-882a-44eec7d799df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.322848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.322807    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.322996 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.322859    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.322996 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.322909    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddfda63-2274-4793-bb2a-e99d52000804-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.322996 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.322953    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddfda63-2274-4793-bb2a-e99d52000804-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323003    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzqr\" (UniqueName: \"kubernetes.io/projected/5ddfda63-2274-4793-bb2a-e99d52000804-kube-api-access-2gzqr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323034    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323234    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddfda63-2274-4793-bb2a-e99d52000804-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323290    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323436 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323323    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323436 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323330    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323436 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323561 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323502    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323687 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323664    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.323821 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.323804    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddfda63-2274-4793-bb2a-e99d52000804-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.325606 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.325580    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddfda63-2274-4793-bb2a-e99d52000804-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.325853 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.325829    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddfda63-2274-4793-bb2a-e99d52000804-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.380075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.380043    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzqr\" (UniqueName: \"kubernetes.io/projected/5ddfda63-2274-4793-bb2a-e99d52000804-kube-api-access-2gzqr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.380649 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.380629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddfda63-2274-4793-bb2a-e99d52000804-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tgp7n\" (UID: \"5ddfda63-2274-4793-bb2a-e99d52000804\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.390652 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.390635    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:43.459979 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.459945    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:43.805995 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:37:43.805962    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0bbf59_4728_4a9a_882a_44eec7d799df.slice/crio-de4e7108decdaa7ff4031e42e63a80fbea238e54e116366a31f9e72f2c3ea2ca WatchSource:0}: Error finding container de4e7108decdaa7ff4031e42e63a80fbea238e54e116366a31f9e72f2c3ea2ca: Status 404 returned error can't find the container with id de4e7108decdaa7ff4031e42e63a80fbea238e54e116366a31f9e72f2c3ea2ca
Nov 02 22:37:43.808126 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.808081    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"]
Nov 02 22:37:43.808607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.808582    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:37:43.808669 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.808637    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:37:43.808707 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.808688    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:37:43.855100 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:37:43.855047    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddfda63_2274_4793_bb2a_e99d52000804.slice/crio-2e4ddcb787342dc40f4b52aa34e1ce5a8807d4644a0b50867c289250f2ccaf1b WatchSource:0}: Error finding container 2e4ddcb787342dc40f4b52aa34e1ce5a8807d4644a0b50867c289250f2ccaf1b: Status 404 returned error can't find the container with id 2e4ddcb787342dc40f4b52aa34e1ce5a8807d4644a0b50867c289250f2ccaf1b
Nov 02 22:37:43.857589 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.857568    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"]
Nov 02 22:37:43.857689 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.857622    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:37:43.857729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.857686    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:37:43.857729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:43.857713    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:37:44.117928 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:37:44.117845    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:37:44.469542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:44.469415    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" event={"ID":"5ddfda63-2274-4793-bb2a-e99d52000804","Type":"ContainerStarted","Data":"aec05512ed4f7ca29a11c79c43aca0bc9548ff169c27bfad48bccdb5d634cdb3"}
Nov 02 22:37:44.469542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:44.469468    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" event={"ID":"5ddfda63-2274-4793-bb2a-e99d52000804","Type":"ContainerStarted","Data":"2e4ddcb787342dc40f4b52aa34e1ce5a8807d4644a0b50867c289250f2ccaf1b"}
Nov 02 22:37:44.470997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:44.470973    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" event={"ID":"2a0bbf59-4728-4a9a-882a-44eec7d799df","Type":"ContainerStarted","Data":"574c48ed1351b3769166804d62f3757118471b053d594aa3f7f9cd1350bc9b62"}
Nov 02 22:37:44.471128 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:44.471016    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" event={"ID":"2a0bbf59-4728-4a9a-882a-44eec7d799df","Type":"ContainerStarted","Data":"de4e7108decdaa7ff4031e42e63a80fbea238e54e116366a31f9e72f2c3ea2ca"}
Nov 02 22:37:44.592471 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:44.592414    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podStartSLOduration=1.592398814 podStartE2EDuration="1.592398814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:44.590594897 +0000 UTC m=+5351.854131213" watchObservedRunningTime="2026-11-02 22:37:44.592398814 +0000 UTC m=+5351.855935130"
Nov 02 22:37:44.716131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:44.715387    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podStartSLOduration=1.715363822 podStartE2EDuration="1.715363822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:44.712732618 +0000 UTC m=+5351.976268946" watchObservedRunningTime="2026-11-02 22:37:44.715363822 +0000 UTC m=+5351.978900143"
Nov 02 22:37:45.354349 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:45.354312    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:37:45.391058 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:45.391031    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:45.392420 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:45.392399    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused" start-of-body=
Nov 02 22:37:45.392474 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:45.392443    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused"
Nov 02 22:37:45.460993 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:45.460952    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:45.462129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:45.462092    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 02 22:37:45.462232 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:45.462152    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 02 22:37:46.391163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:46.391095    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused" start-of-body=
Nov 02 22:37:46.391592 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:46.391172    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused"
Nov 02 22:37:46.461061 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:46.461018    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 02 22:37:46.461264 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:46.461082    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 02 22:37:47.391569 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:47.391523    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused" start-of-body=
Nov 02 22:37:47.391981 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:47.391584    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused"
Nov 02 22:37:47.461964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:47.461926    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 02 22:37:47.462161 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:47.461996    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 02 22:37:48.053708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:48.053674    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"]
Nov 02 22:37:48.123048 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:48.123011    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4e8959-l4gc5"]
Nov 02 22:37:48.391498 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:48.391418    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused" start-of-body=
Nov 02 22:37:48.391726 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:48.391697    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused"
Nov 02 22:37:48.461305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:48.461266    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 02 22:37:48.461490 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:48.461323    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 02 22:37:48.941137 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:48.935262    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031695c6-2787-46f8-9f0f-05c289f7f583" path="/var/lib/kubelet/pods/031695c6-2787-46f8-9f0f-05c289f7f583/volumes"
Nov 02 22:37:49.391217 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:49.391178    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused" start-of-body=
Nov 02 22:37:49.391406 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:49.391230    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused"
Nov 02 22:37:49.461290 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:49.461252    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 02 22:37:49.461694 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:49.461300    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 02 22:37:50.391495 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:50.391454    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused" start-of-body=
Nov 02 22:37:50.391685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:50.391560    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": dial tcp 10.133.1.122:15021: connect: connection refused"
Nov 02 22:37:50.461533 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:50.461499    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 02 22:37:50.461909 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:50.461551    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 02 22:37:52.392274 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:52.392228    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:37:52.392804 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:52.392301    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:37:52.461357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:52.461322    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:37:52.461645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:52.461603    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:37:53.443268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:53.443226    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:37:53.443910 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:53.443863    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:37:53.521130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:53.510201    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:37:53.521130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:53.510279    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:37:54.485297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.485254    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:37:54.485715 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.485311    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:37:54.551314 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.551280    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:37:54.551590 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.551564    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:37:54.825337 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.825304    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:54.825810 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.825789    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:54.826577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.826557    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n"
Nov 02 22:37:54.826801 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.826788    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:54.827052 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.827032    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:54.827651 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:54.827634    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp"
Nov 02 22:37:55.735718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:55.735687    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:37:59.045683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:59.045653    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"]
Nov 02 22:37:59.061645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:37:59.061618    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a371ee65-5g8h6"]
Nov 02 22:38:00.911008 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:38:00.910972    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc573ba-957e-4d34-beca-4a1bea38e962" path="/var/lib/kubelet/pods/6cc573ba-957e-4d34-beca-4a1bea38e962/volumes"
Nov 02 22:38:04.538523 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:38:04.538487    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:38:13.959434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:38:13.959396    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:38:14.111857 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:38:14.111757    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:38:23.851725 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:38:23.851687    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:38:37.096438 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:38:37.096410    4528 scope.go:122] "RemoveContainer" containerID="dd266a46500abb4b0c71cbad59bef65de86e1ca407df28d032c356726c1d969c"
Nov 02 22:38:37.104380 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:38:37.104360    4528 scope.go:122] "RemoveContainer" containerID="d047a7ab0ec672fc5c76dd5cf3a1096002d81e0f707130c5cd3e5f531aa5bbdb"
Nov 02 22:38:42.387729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:38:42.387696    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:38:44.176551 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:38:44.176505    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:38:56.770196 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:38:56.770159    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:39:14.111697 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:39:14.111662    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:39:28.152845 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:28.152804    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:39:37.125739 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:37.125708    4528 scope.go:122] "RemoveContainer" containerID="1cfd1512c7e1685e36212dd525bc7889f3526c4fa45477e5fd2540bef2352a6f"
Nov 02 22:39:37.133806 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:37.133789    4528 scope.go:122] "RemoveContainer" containerID="8318244b8c548c6ff353a9d21aeac562fdf759cb8e91b1849aa1d833ee695282"
Nov 02 22:39:38.852807 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.852762    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"]
Nov 02 22:39:38.859827 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.859542    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:38.913075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.913044    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"]
Nov 02 22:39:38.930803 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.930766    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsc8s\" (UniqueName: \"kubernetes.io/projected/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-kube-api-access-nsc8s\") pod \"maas-api-revoke-sub-1fe07478-d8gw4\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:38.930965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.930811    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe07478-d8gw4\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:38.930965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.930916    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe07478-d8gw4\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:38.960007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.959984    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:39:38.961770 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.961752    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:39:38.961875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.961768    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:39:38.961875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:38.961771    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:39:39.031765 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:39.031735    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe07478-d8gw4\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:39.031963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:39.031778    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe07478-d8gw4\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:39.031963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:39.031852    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsc8s\" (UniqueName: \"kubernetes.io/projected/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-kube-api-access-nsc8s\") pod \"maas-api-revoke-sub-1fe07478-d8gw4\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:39.032528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:39.032502    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe07478-d8gw4\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:39.033994 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:39.033975    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe07478-d8gw4\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:39.088490 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:39.088462    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsc8s\" (UniqueName: \"kubernetes.io/projected/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-kube-api-access-nsc8s\") pod \"maas-api-revoke-sub-1fe07478-d8gw4\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:39.172470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:39.172377    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:39.536362 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:39:39.536326    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4c2d48_c3d5_416d_a3cd_b46258c63ced.slice/crio-545850b00b99242660e015881c00c3005ab7653051f2157eb038865dcbba2d63 WatchSource:0}: Error finding container 545850b00b99242660e015881c00c3005ab7653051f2157eb038865dcbba2d63: Status 404 returned error can't find the container with id 545850b00b99242660e015881c00c3005ab7653051f2157eb038865dcbba2d63
Nov 02 22:39:39.538304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:39.538280    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"]
Nov 02 22:39:40.051429 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:40.051398    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4" event={"ID":"4f4c2d48-c3d5-416d-a3cd-b46258c63ced","Type":"ContainerStarted","Data":"89bad85f1585f7d7ce070c6e2da827145626c200937c53196b6fcce8d4250c20"}
Nov 02 22:39:40.051797 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:40.051452    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4" event={"ID":"4f4c2d48-c3d5-416d-a3cd-b46258c63ced","Type":"ContainerStarted","Data":"545850b00b99242660e015881c00c3005ab7653051f2157eb038865dcbba2d63"}
Nov 02 22:39:40.164892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:40.164844    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4" podStartSLOduration=2.164832637 podStartE2EDuration="2.164832637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:40.163872131 +0000 UTC m=+5467.427408448" watchObservedRunningTime="2026-11-02 22:39:40.164832637 +0000 UTC m=+5467.428368954"
Nov 02 22:39:41.057372 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:41.057331    4528 generic.go:356] "Generic (PLEG): container finished" podID="4f4c2d48-c3d5-416d-a3cd-b46258c63ced" containerID="89bad85f1585f7d7ce070c6e2da827145626c200937c53196b6fcce8d4250c20" exitCode=0
Nov 02 22:39:41.057762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:41.057413    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4" event={"ID":"4f4c2d48-c3d5-416d-a3cd-b46258c63ced","Type":"ContainerDied","Data":"89bad85f1585f7d7ce070c6e2da827145626c200937c53196b6fcce8d4250c20"}
Nov 02 22:39:42.194153 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.194129    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:42.261364 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.261340    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-kube-api-access-nsc8s\" (UniqueName: \"kubernetes.io/projected/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-kube-api-access-nsc8s\") pod \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") "
Nov 02 22:39:42.261504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.261378    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-xks-ca\") pod \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") "
Nov 02 22:39:42.261504 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.261471    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-openshift-ca\") pod \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\" (UID: \"4f4c2d48-c3d5-416d-a3cd-b46258c63ced\") "
Nov 02 22:39:42.261777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.261758    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-openshift-ca" pod "4f4c2d48-c3d5-416d-a3cd-b46258c63ced" (UID: "4f4c2d48-c3d5-416d-a3cd-b46258c63ced"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:39:42.263374 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.263352    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-xks-ca" pod "4f4c2d48-c3d5-416d-a3cd-b46258c63ced" (UID: "4f4c2d48-c3d5-416d-a3cd-b46258c63ced"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:39:42.263476 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.263426    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-kube-api-access-nsc8s" pod "4f4c2d48-c3d5-416d-a3cd-b46258c63ced" (UID: "4f4c2d48-c3d5-416d-a3cd-b46258c63ced"). InnerVolumeSpecName "kube-api-access-nsc8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:39:42.362850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.362778    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:42.362850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.362803    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsc8s\" (UniqueName: \"kubernetes.io/projected/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-kube-api-access-nsc8s\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:42.362850 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:42.362817    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4c2d48-c3d5-416d-a3cd-b46258c63ced-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:39:43.067216 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:43.067181    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4" event={"ID":"4f4c2d48-c3d5-416d-a3cd-b46258c63ced","Type":"ContainerDied","Data":"545850b00b99242660e015881c00c3005ab7653051f2157eb038865dcbba2d63"}
Nov 02 22:39:43.067216 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:43.067211    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"
Nov 02 22:39:43.067427 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:43.067230    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="545850b00b99242660e015881c00c3005ab7653051f2157eb038865dcbba2d63"
Nov 02 22:39:44.111990 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:39:44.111956    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:39:51.951033 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:51.950998    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:39:57.031270 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.031236    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"]
Nov 02 22:39:57.031794 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.031774    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4f4c2d48-c3d5-416d-a3cd-b46258c63ced" containerName="revoke-keys"
Nov 02 22:39:57.031889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.031796    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4c2d48-c3d5-416d-a3cd-b46258c63ced" containerName="revoke-keys"
Nov 02 22:39:57.031951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.031927    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4c2d48-c3d5-416d-a3cd-b46258c63ced" containerName="revoke-keys"
Nov 02 22:39:57.035057 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.035038    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.097597 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.097564    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrw88\" (UniqueName: \"kubernetes.io/projected/2bbf0d00-8b7b-4f6a-992e-684686f0db07-kube-api-access-lrw88\") pod \"maas-api-revoke-sub-c08e6842-gqj7s\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.097791 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.097624    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08e6842-gqj7s\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.097791 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.097759    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08e6842-gqj7s\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.140036 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.140001    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:39:57.140036 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.140001    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:39:57.140036 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.140034    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:39:57.140342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.140148    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:39:57.141741 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.141722    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"]
Nov 02 22:39:57.198386 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.198356    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08e6842-gqj7s\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.198386 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.198390    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrw88\" (UniqueName: \"kubernetes.io/projected/2bbf0d00-8b7b-4f6a-992e-684686f0db07-kube-api-access-lrw88\") pod \"maas-api-revoke-sub-c08e6842-gqj7s\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.198583 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.198416    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08e6842-gqj7s\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.198963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.198943    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08e6842-gqj7s\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.200899 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.200878    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08e6842-gqj7s\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.255211 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.255185    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrw88\" (UniqueName: \"kubernetes.io/projected/2bbf0d00-8b7b-4f6a-992e-684686f0db07-kube-api-access-lrw88\") pod \"maas-api-revoke-sub-c08e6842-gqj7s\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.351062 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.350969    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:39:57.728792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:57.728754    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"]
Nov 02 22:39:58.132588 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:58.132554    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s" event={"ID":"2bbf0d00-8b7b-4f6a-992e-684686f0db07","Type":"ContainerStarted","Data":"88ee0b6a6658ca737b2c1e264aeb314620b7f7e62c335d49019be33c49032254"}
Nov 02 22:39:58.133092 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:58.132620    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s" event={"ID":"2bbf0d00-8b7b-4f6a-992e-684686f0db07","Type":"ContainerStarted","Data":"b1c7ad084911ce37cb944b231fb8161ca71493ff377fc0dea41004124d5bb7d3"}
Nov 02 22:39:58.254133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:58.252028    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s" podStartSLOduration=2.252010539 podStartE2EDuration="2.252010539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:58.25128023 +0000 UTC m=+5485.514816547" watchObservedRunningTime="2026-11-02 22:39:58.252010539 +0000 UTC m=+5485.515546856"
Nov 02 22:39:59.154135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:59.149510    4528 generic.go:356] "Generic (PLEG): container finished" podID="2bbf0d00-8b7b-4f6a-992e-684686f0db07" containerID="88ee0b6a6658ca737b2c1e264aeb314620b7f7e62c335d49019be33c49032254" exitCode=0
Nov 02 22:39:59.154135 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:39:59.149642    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s" event={"ID":"2bbf0d00-8b7b-4f6a-992e-684686f0db07","Type":"ContainerDied","Data":"88ee0b6a6658ca737b2c1e264aeb314620b7f7e62c335d49019be33c49032254"}
Nov 02 22:40:00.287400 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.287377    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:40:00.379693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.379664    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-openshift-ca\") pod \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") "
Nov 02 22:40:00.379875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.379848    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbf0d00-8b7b-4f6a-992e-684686f0db07-kube-api-access-lrw88\" (UniqueName: \"kubernetes.io/projected/2bbf0d00-8b7b-4f6a-992e-684686f0db07-kube-api-access-lrw88\") pod \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") "
Nov 02 22:40:00.379964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.379951    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-xks-ca\") pod \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\" (UID: \"2bbf0d00-8b7b-4f6a-992e-684686f0db07\") "
Nov 02 22:40:00.380030 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.379968    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-openshift-ca" pod "2bbf0d00-8b7b-4f6a-992e-684686f0db07" (UID: "2bbf0d00-8b7b-4f6a-992e-684686f0db07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:40:00.380255 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.380238    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:40:00.381792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.381773    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-xks-ca" pod "2bbf0d00-8b7b-4f6a-992e-684686f0db07" (UID: "2bbf0d00-8b7b-4f6a-992e-684686f0db07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:40:00.381851 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.381829    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbf0d00-8b7b-4f6a-992e-684686f0db07-kube-api-access-lrw88" pod "2bbf0d00-8b7b-4f6a-992e-684686f0db07" (UID: "2bbf0d00-8b7b-4f6a-992e-684686f0db07"). InnerVolumeSpecName "kube-api-access-lrw88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:40:00.481430 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.481354    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrw88\" (UniqueName: \"kubernetes.io/projected/2bbf0d00-8b7b-4f6a-992e-684686f0db07-kube-api-access-lrw88\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:40:00.481430 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:00.481382    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbf0d00-8b7b-4f6a-992e-684686f0db07-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:40:01.160497 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:01.160467    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"
Nov 02 22:40:01.160669 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:01.160469    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s" event={"ID":"2bbf0d00-8b7b-4f6a-992e-684686f0db07","Type":"ContainerDied","Data":"b1c7ad084911ce37cb944b231fb8161ca71493ff377fc0dea41004124d5bb7d3"}
Nov 02 22:40:01.160669 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:01.160595    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1c7ad084911ce37cb944b231fb8161ca71493ff377fc0dea41004124d5bb7d3"
Nov 02 22:40:03.475098 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:03.475067    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:40:09.927844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.927814    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"]
Nov 02 22:40:09.928268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.928173    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2bbf0d00-8b7b-4f6a-992e-684686f0db07" containerName="revoke-keys"
Nov 02 22:40:09.928268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.928185    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbf0d00-8b7b-4f6a-992e-684686f0db07" containerName="revoke-keys"
Nov 02 22:40:09.928268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.928256    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbf0d00-8b7b-4f6a-992e-684686f0db07" containerName="revoke-keys"
Nov 02 22:40:09.931630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.931610    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:09.980745 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.980701    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfvqx\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-kube-api-access-rfvqx\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:09.981045 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.981017    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-credential-socket\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:09.981285 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.981262    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-envoy\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:09.981507 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.981488    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-certs\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:09.981665 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.981645    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-data\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:09.981855 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.981835    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-podinfo\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:09.982102 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.982082    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-token\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:09.982356 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.982338    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-socket\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:09.983595 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:09.982512    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istiod-ca-cert\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.002548 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.002511    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-6faf18-openshift-default-dockercfg-rrzqk\""
Nov 02 22:40:10.036629 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.036591    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"]
Nov 02 22:40:10.088356 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.088318    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-certs\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.088648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.088621    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-data\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.088792 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.088773    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-podinfo\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.088920 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.088895    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-token\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.089078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.089057    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-data\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.089225 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.089208    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-socket\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.089356 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.089340    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istiod-ca-cert\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.089470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.089454    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfvqx\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-kube-api-access-rfvqx\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.089607 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.089587    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-credential-socket\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.089693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.089679    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-envoy\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.091512 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.091457    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-socket\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.091746 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.088804    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-certs\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.092725 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.092699    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istiod-ca-cert\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.093054 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.093032    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-credential-socket\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.097851 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.097827    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-envoy\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.098874 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.098846    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-podinfo\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.150219 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.150184    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-token\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.150512 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.150496    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfvqx\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-kube-api-access-rfvqx\") pod \"e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") " pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.244433 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.244356    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:10.671167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.671138    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"]
Nov 02 22:40:10.671799 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:40:10.671772    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6b3597_121b_4f4b_aab9_031a2124ff0f.slice/crio-d2f32704af75329c31a1dcc9cfc7cbfe05b2ebdd38f2847acc0e2b3621f16f9c WatchSource:0}: Error finding container d2f32704af75329c31a1dcc9cfc7cbfe05b2ebdd38f2847acc0e2b3621f16f9c: Status 404 returned error can't find the container with id d2f32704af75329c31a1dcc9cfc7cbfe05b2ebdd38f2847acc0e2b3621f16f9c
Nov 02 22:40:10.674544 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.674506    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:40:10.674636 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.674589    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:40:10.674636 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:10.674628    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:40:11.233138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:11.233064    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" event={"ID":"2e6b3597-121b-4f4b-aab9-031a2124ff0f","Type":"ContainerStarted","Data":"9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d"}
Nov 02 22:40:11.233654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:11.233147    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" event={"ID":"2e6b3597-121b-4f4b-aab9-031a2124ff0f","Type":"ContainerStarted","Data":"d2f32704af75329c31a1dcc9cfc7cbfe05b2ebdd38f2847acc0e2b3621f16f9c"}
Nov 02 22:40:11.254674 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:11.254647    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:11.256798 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:11.256772    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 02 22:40:11.257009 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:11.256982    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 02 22:40:11.366339 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:11.366277    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" podStartSLOduration=2.366256784 podStartE2EDuration="2.366256784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:40:11.363544135 +0000 UTC m=+5498.627080488" watchObservedRunningTime="2026-11-02 22:40:11.366256784 +0000 UTC m=+5498.629793102"
Nov 02 22:40:12.271725 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:12.271692    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:13.254329 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:13.254296    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:13.255273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:13.255253    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:14.112432 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:40:14.112399    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:40:17.752731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.752698    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"]
Nov 02 22:40:17.756325 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.756309    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:17.812880 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.812842    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"]
Nov 02 22:40:17.830476 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.830448    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcgb\" (UniqueName: \"kubernetes.io/projected/d75bc0c3-a857-40fc-ba96-50483c3bf532-kube-api-access-7gcgb\") pod \"payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg\" (UID: \"d75bc0c3-a857-40fc-ba96-50483c3bf532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:17.830622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.830482    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75bc0c3-a857-40fc-ba96-50483c3bf532-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg\" (UID: \"d75bc0c3-a857-40fc-ba96-50483c3bf532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:17.863975 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.863944    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f5b1e307-dockercfg-mr9k8\""
Nov 02 22:40:17.863975 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.863945    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6faf18\""
Nov 02 22:40:17.912143 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.912097    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"]
Nov 02 22:40:17.915622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.915602    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:17.931052 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.931030    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcgb\" (UniqueName: \"kubernetes.io/projected/d75bc0c3-a857-40fc-ba96-50483c3bf532-kube-api-access-7gcgb\") pod \"payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg\" (UID: \"d75bc0c3-a857-40fc-ba96-50483c3bf532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:17.931226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.931078    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75bc0c3-a857-40fc-ba96-50483c3bf532-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg\" (UID: \"d75bc0c3-a857-40fc-ba96-50483c3bf532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:17.931226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.931167    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8k4\" (UniqueName: \"kubernetes.io/projected/455e8421-a549-498c-917e-361d5d3d7144-kube-api-access-ks8k4\") pod \"payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6\" (UID: \"455e8421-a549-498c-917e-361d5d3d7144\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:17.931226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.931209    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455e8421-a549-498c-917e-361d5d3d7144-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6\" (UID: \"455e8421-a549-498c-917e-361d5d3d7144\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:17.931720 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.931700    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75bc0c3-a857-40fc-ba96-50483c3bf532-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg\" (UID: \"d75bc0c3-a857-40fc-ba96-50483c3bf532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:17.973928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.973904    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"]
Nov 02 22:40:17.987502 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:17.987476    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcgb\" (UniqueName: \"kubernetes.io/projected/d75bc0c3-a857-40fc-ba96-50483c3bf532-kube-api-access-7gcgb\") pod \"payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg\" (UID: \"d75bc0c3-a857-40fc-ba96-50483c3bf532\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:18.031675 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:18.031648    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8k4\" (UniqueName: \"kubernetes.io/projected/455e8421-a549-498c-917e-361d5d3d7144-kube-api-access-ks8k4\") pod \"payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6\" (UID: \"455e8421-a549-498c-917e-361d5d3d7144\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:18.031806 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:18.031691    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455e8421-a549-498c-917e-361d5d3d7144-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6\" (UID: \"455e8421-a549-498c-917e-361d5d3d7144\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:18.032392 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:18.032372    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455e8421-a549-498c-917e-361d5d3d7144-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6\" (UID: \"455e8421-a549-498c-917e-361d5d3d7144\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:18.069078 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:18.069055    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:18.088226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:18.088207    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8k4\" (UniqueName: \"kubernetes.io/projected/455e8421-a549-498c-917e-361d5d3d7144-kube-api-access-ks8k4\") pod \"payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6\" (UID: \"455e8421-a549-498c-917e-361d5d3d7144\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:18.228192 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:18.228162    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:18.433008 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:40:18.432935    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75bc0c3_a857_40fc_ba96_50483c3bf532.slice/crio-a7421c75503b35aee38ee8d246f5c8174b1499209540cc20dd1b58d26564d7b2 WatchSource:0}: Error finding container a7421c75503b35aee38ee8d246f5c8174b1499209540cc20dd1b58d26564d7b2: Status 404 returned error can't find the container with id a7421c75503b35aee38ee8d246f5c8174b1499209540cc20dd1b58d26564d7b2
Nov 02 22:40:18.434585 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:18.434532    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"]
Nov 02 22:40:18.591899 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:40:18.591866    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455e8421_a549_498c_917e_361d5d3d7144.slice/crio-644bfd165411d3eed6f3543c458481c094d93b644b300fbed490ae641313917c WatchSource:0}: Error finding container 644bfd165411d3eed6f3543c458481c094d93b644b300fbed490ae641313917c: Status 404 returned error can't find the container with id 644bfd165411d3eed6f3543c458481c094d93b644b300fbed490ae641313917c
Nov 02 22:40:18.593710 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:18.593671    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"]
Nov 02 22:40:19.290460 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:19.290420    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6" event={"ID":"455e8421-a549-498c-917e-361d5d3d7144","Type":"ContainerStarted","Data":"f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a"}
Nov 02 22:40:19.290952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:19.290599    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:19.290952 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:19.290646    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6" event={"ID":"455e8421-a549-498c-917e-361d5d3d7144","Type":"ContainerStarted","Data":"644bfd165411d3eed6f3543c458481c094d93b644b300fbed490ae641313917c"}
Nov 02 22:40:19.292057 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:19.292033    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg" event={"ID":"d75bc0c3-a857-40fc-ba96-50483c3bf532","Type":"ContainerStarted","Data":"f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6"}
Nov 02 22:40:19.292175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:19.292081    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:19.292175 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:19.292097    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg" event={"ID":"d75bc0c3-a857-40fc-ba96-50483c3bf532","Type":"ContainerStarted","Data":"a7421c75503b35aee38ee8d246f5c8174b1499209540cc20dd1b58d26564d7b2"}
Nov 02 22:40:19.405206 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:19.405150    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6" podStartSLOduration=2.405132285 podStartE2EDuration="2.405132285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:40:19.40447487 +0000 UTC m=+5506.668011187" watchObservedRunningTime="2026-11-02 22:40:19.405132285 +0000 UTC m=+5506.668668604"
Nov 02 22:40:19.528671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:19.528614    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg" podStartSLOduration=2.5285975670000003 podStartE2EDuration="2.528597567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:40:19.526076859 +0000 UTC m=+5506.789613199" watchObservedRunningTime="2026-11-02 22:40:19.528597567 +0000 UTC m=+5506.792133885"
Nov 02 22:40:29.655845 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:29.655800    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:40:30.299518 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:30.299490    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:40:30.299679 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:30.299554    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:40:43.255579 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:43.255542    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 02 22:40:43.255997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:43.255614    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 02 22:40:43.400737 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:43.400645    4528 generic.go:356] "Generic (PLEG): container finished" podID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerID="9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d" exitCode=0
Nov 02 22:40:43.400737 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:43.400685    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" event={"ID":"2e6b3597-121b-4f4b-aab9-031a2124ff0f","Type":"ContainerDied","Data":"9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d"}
Nov 02 22:40:43.401010 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:43.400995    4528 scope.go:122] "RemoveContainer" containerID="9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d"
Nov 02 22:40:43.404730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:43.404700    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:40:43.404794 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:43.404772    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:40:43.404836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:43.404824    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:40:44.130837 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:40:44.130796    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:40:44.429065 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:44.428955    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" event={"ID":"2e6b3597-121b-4f4b-aab9-031a2124ff0f","Type":"ContainerStarted","Data":"b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b"}
Nov 02 22:40:45.244901 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:45.244866    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:45.245094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:45.245020    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 02 22:40:45.245094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:45.245069    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 02 22:40:46.245753 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:46.245718    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 02 22:40:46.246183 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:46.245763    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 02 22:40:46.969129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:46.968208    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:40:47.244837 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:47.244760    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 02 22:40:47.244837 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:47.244820    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 02 22:40:48.245368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:48.245327    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 02 22:40:48.245824 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:48.245389    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 02 22:40:49.702657 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:49.702622    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:49.703384 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:49.702940    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:40:49.703956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:40:49.703935    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:41:14.112220 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:41:14.112179    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:41:14.635986 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:41:14.635947    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:41:40.175889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:41:40.175852    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:41:44.111715 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:41:44.111678    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:41:45.237774 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:41:45.237742    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"]
Nov 02 22:41:45.345928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:41:45.345893    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"]
Nov 02 22:41:50.238590 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:41:50.238548    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6" podUID="455e8421-a549-498c-917e-361d5d3d7144" containerName="payload-processing" containerID="cri-o://f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a" gracePeriod=25
Nov 02 22:41:50.298156 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:41:50.298103    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6" podUID="455e8421-a549-498c-917e-361d5d3d7144" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:41:50.347147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:41:50.347081    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg" podUID="d75bc0c3-a857-40fc-ba96-50483c3bf532" containerName="payload-pre-processing" containerID="cri-o://f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6" gracePeriod=25
Nov 02 22:41:53.563011 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:41:53.562973    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:42:00.298373 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:00.298326    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6" podUID="455e8421-a549-498c-917e-361d5d3d7144" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:42:00.298373 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:00.298351    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg" podUID="d75bc0c3-a857-40fc-ba96-50483c3bf532" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:42:10.297499 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:42:10.297472    4528 logging.go:55] [core] [Channel #22131 SubChannel #22133]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.128:50052", ServerName: "10.133.1.128:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.128:50052: connect: connection refused"
Nov 02 22:42:10.298005 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.297978    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg" podUID="d75bc0c3-a857-40fc-ba96-50483c3bf532" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 22:42:10.485006 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.484972    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:42:10.561621 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.561593    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455e8421-a549-498c-917e-361d5d3d7144-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455e8421-a549-498c-917e-361d5d3d7144-plugins-config-volume\") pod \"455e8421-a549-498c-917e-361d5d3d7144\" (UID: \"455e8421-a549-498c-917e-361d5d3d7144\") "
Nov 02 22:42:10.561805 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.561652    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455e8421-a549-498c-917e-361d5d3d7144-kube-api-access-ks8k4\" (UniqueName: \"kubernetes.io/projected/455e8421-a549-498c-917e-361d5d3d7144-kube-api-access-ks8k4\") pod \"455e8421-a549-498c-917e-361d5d3d7144\" (UID: \"455e8421-a549-498c-917e-361d5d3d7144\") "
Nov 02 22:42:10.561965 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.561935    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455e8421-a549-498c-917e-361d5d3d7144-plugins-config-volume" pod "455e8421-a549-498c-917e-361d5d3d7144" (UID: "455e8421-a549-498c-917e-361d5d3d7144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:42:10.563772 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.563746    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455e8421-a549-498c-917e-361d5d3d7144-kube-api-access-ks8k4" pod "455e8421-a549-498c-917e-361d5d3d7144" (UID: "455e8421-a549-498c-917e-361d5d3d7144"). InnerVolumeSpecName "kube-api-access-ks8k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:42:10.579204 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.579178    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:42:10.662311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.662281    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75bc0c3-a857-40fc-ba96-50483c3bf532-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75bc0c3-a857-40fc-ba96-50483c3bf532-plugins-config-volume\") pod \"d75bc0c3-a857-40fc-ba96-50483c3bf532\" (UID: \"d75bc0c3-a857-40fc-ba96-50483c3bf532\") "
Nov 02 22:42:10.662444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.662364    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75bc0c3-a857-40fc-ba96-50483c3bf532-kube-api-access-7gcgb\" (UniqueName: \"kubernetes.io/projected/d75bc0c3-a857-40fc-ba96-50483c3bf532-kube-api-access-7gcgb\") pod \"d75bc0c3-a857-40fc-ba96-50483c3bf532\" (UID: \"d75bc0c3-a857-40fc-ba96-50483c3bf532\") "
Nov 02 22:42:10.662582 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.662570    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455e8421-a549-498c-917e-361d5d3d7144-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:10.662630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.662584    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8k4\" (UniqueName: \"kubernetes.io/projected/455e8421-a549-498c-917e-361d5d3d7144-kube-api-access-ks8k4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:10.662668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.662644    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75bc0c3-a857-40fc-ba96-50483c3bf532-plugins-config-volume" pod "d75bc0c3-a857-40fc-ba96-50483c3bf532" (UID: "d75bc0c3-a857-40fc-ba96-50483c3bf532"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:42:10.664307 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.664288    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75bc0c3-a857-40fc-ba96-50483c3bf532-kube-api-access-7gcgb" pod "d75bc0c3-a857-40fc-ba96-50483c3bf532" (UID: "d75bc0c3-a857-40fc-ba96-50483c3bf532"). InnerVolumeSpecName "kube-api-access-7gcgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:42:10.763283 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.763257    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75bc0c3-a857-40fc-ba96-50483c3bf532-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:10.763283 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.763281    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gcgb\" (UniqueName: \"kubernetes.io/projected/d75bc0c3-a857-40fc-ba96-50483c3bf532-kube-api-access-7gcgb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:10.832997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.832924    4528 generic.go:356] "Generic (PLEG): container finished" podID="d75bc0c3-a857-40fc-ba96-50483c3bf532" containerID="f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6" exitCode=0
Nov 02 22:42:10.832997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.832966    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"
Nov 02 22:42:10.833211 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.833020    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg" event={"ID":"d75bc0c3-a857-40fc-ba96-50483c3bf532","Type":"ContainerDied","Data":"f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6"}
Nov 02 22:42:10.833211 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.833070    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg" event={"ID":"d75bc0c3-a857-40fc-ba96-50483c3bf532","Type":"ContainerDied","Data":"a7421c75503b35aee38ee8d246f5c8174b1499209540cc20dd1b58d26564d7b2"}
Nov 02 22:42:10.833211 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.833099    4528 scope.go:122] "RemoveContainer" containerID="f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6"
Nov 02 22:42:10.834429 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.834396    4528 generic.go:356] "Generic (PLEG): container finished" podID="455e8421-a549-498c-917e-361d5d3d7144" containerID="f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a" exitCode=0
Nov 02 22:42:10.834530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.834448    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6" event={"ID":"455e8421-a549-498c-917e-361d5d3d7144","Type":"ContainerDied","Data":"f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a"}
Nov 02 22:42:10.834530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.834487    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6" event={"ID":"455e8421-a549-498c-917e-361d5d3d7144","Type":"ContainerDied","Data":"644bfd165411d3eed6f3543c458481c094d93b644b300fbed490ae641313917c"}
Nov 02 22:42:10.834530 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.834494    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"
Nov 02 22:42:10.841738 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.841722    4528 scope.go:122] "RemoveContainer" containerID="f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6"
Nov 02 22:42:10.842015 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:42:10.841989    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6\": container with ID starting with f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6 not found: ID does not exist" containerID="f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6"
Nov 02 22:42:10.842089 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.842025    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6"} err="failed to get container status \"f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6\": rpc error: code = NotFound desc = could not find container \"f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6\": container with ID starting with f00a133c475a5e3cc1ef0124df256b1799fddd5d2697485c147ac8160e2271f6 not found: ID does not exist"
Nov 02 22:42:10.842089 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.842047    4528 scope.go:122] "RemoveContainer" containerID="f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a"
Nov 02 22:42:10.849763 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.849745    4528 scope.go:122] "RemoveContainer" containerID="f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a"
Nov 02 22:42:10.850027 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:42:10.850000    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a\": container with ID starting with f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a not found: ID does not exist" containerID="f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a"
Nov 02 22:42:10.850077 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:10.850035    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a"} err="failed to get container status \"f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a\": rpc error: code = NotFound desc = could not find container \"f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a\": container with ID starting with f8ea573ca36f05fdf89528b301c78cc30f43c42f8fcbcd33b1a357bc1792e45a not found: ID does not exist"
Nov 02 22:42:11.003947 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:11.003917    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"]
Nov 02 22:42:11.009892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:11.009868    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6"]
Nov 02 22:42:11.117997 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:11.117932    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"]
Nov 02 22:42:11.123030 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:11.123003    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6faf18-6b655f6xpg"]
Nov 02 22:42:11.297651 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:11.297611    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6faf18-5cc6fc56f895n6" podUID="455e8421-a549-498c-917e-361d5d3d7144" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.128:50052\" within 1s: context deadline exceeded"
Nov 02 22:42:12.912508 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:12.912476    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455e8421-a549-498c-917e-361d5d3d7144" path="/var/lib/kubelet/pods/455e8421-a549-498c-917e-361d5d3d7144/volumes"
Nov 02 22:42:12.912892 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:12.912838    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75bc0c3-a857-40fc-ba96-50483c3bf532" path="/var/lib/kubelet/pods/d75bc0c3-a857-40fc-ba96-50483c3bf532/volumes"
Nov 02 22:42:14.112583 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:42:14.112546    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:42:23.383644 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:23.383609    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"]
Nov 02 22:42:23.384075 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:23.383834    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy" containerID="cri-o://b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b" gracePeriod=30
Nov 02 22:42:28.630963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.630936    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:42:28.732273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732214    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-envoy\") pod \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") "
Nov 02 22:42:28.732273 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732252    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-socket\") pod \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") "
Nov 02 22:42:28.732441 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732310    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istiod-ca-cert\") pod \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") "
Nov 02 22:42:28.732441 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732351    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-kube-api-access-rfvqx\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-kube-api-access-rfvqx\") pod \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") "
Nov 02 22:42:28.732441 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732384    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-podinfo\") pod \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") "
Nov 02 22:42:28.732611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732444    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-data\") pod \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") "
Nov 02 22:42:28.732611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732495    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-token\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-token\") pod \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") "
Nov 02 22:42:28.732611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732531    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-certs\") pod \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") "
Nov 02 22:42:28.732611 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732560    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-credential-socket\") pod \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\" (UID: \"2e6b3597-121b-4f4b-aab9-031a2124ff0f\") "
Nov 02 22:42:28.732836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732605    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-socket" pod "2e6b3597-121b-4f4b-aab9-031a2124ff0f" (UID: "2e6b3597-121b-4f4b-aab9-031a2124ff0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:42:28.732836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732722    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-data" pod "2e6b3597-121b-4f4b-aab9-031a2124ff0f" (UID: "2e6b3597-121b-4f4b-aab9-031a2124ff0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:42:28.732836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732749    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istiod-ca-cert" pod "2e6b3597-121b-4f4b-aab9-031a2124ff0f" (UID: "2e6b3597-121b-4f4b-aab9-031a2124ff0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:42:28.732836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732825    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-certs" pod "2e6b3597-121b-4f4b-aab9-031a2124ff0f" (UID: "2e6b3597-121b-4f4b-aab9-031a2124ff0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:42:28.733082 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732946    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:28.733082 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732952    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-credential-socket" pod "2e6b3597-121b-4f4b-aab9-031a2124ff0f" (UID: "2e6b3597-121b-4f4b-aab9-031a2124ff0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:42:28.733082 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732965    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:28.733082 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732979    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:28.733082 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.732990    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:28.734543 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.734524    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-envoy" pod "2e6b3597-121b-4f4b-aab9-031a2124ff0f" (UID: "2e6b3597-121b-4f4b-aab9-031a2124ff0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:42:28.734827 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.734803    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-token" pod "2e6b3597-121b-4f4b-aab9-031a2124ff0f" (UID: "2e6b3597-121b-4f4b-aab9-031a2124ff0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:42:28.735092 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.735068    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-podinfo" pod "2e6b3597-121b-4f4b-aab9-031a2124ff0f" (UID: "2e6b3597-121b-4f4b-aab9-031a2124ff0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:42:28.735209 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.735096    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-kube-api-access-rfvqx" pod "2e6b3597-121b-4f4b-aab9-031a2124ff0f" (UID: "2e6b3597-121b-4f4b-aab9-031a2124ff0f"). InnerVolumeSpecName "kube-api-access-rfvqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:42:28.833784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.833748    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfvqx\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-kube-api-access-rfvqx\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:28.833784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.833783    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:28.833784 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.833792    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:28.833994 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.833802    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:28.833994 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.833810    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6b3597-121b-4f4b-aab9-031a2124ff0f-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:42:28.918416 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.918385    4528 generic.go:356] "Generic (PLEG): container finished" podID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerID="b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b" exitCode=0
Nov 02 22:42:28.918578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.918453    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"
Nov 02 22:42:28.918578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.918473    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" event={"ID":"2e6b3597-121b-4f4b-aab9-031a2124ff0f","Type":"ContainerDied","Data":"b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b"}
Nov 02 22:42:28.918578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.918521    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr" event={"ID":"2e6b3597-121b-4f4b-aab9-031a2124ff0f","Type":"ContainerDied","Data":"d2f32704af75329c31a1dcc9cfc7cbfe05b2ebdd38f2847acc0e2b3621f16f9c"}
Nov 02 22:42:28.918578 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.918548    4528 scope.go:122] "RemoveContainer" containerID="b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b"
Nov 02 22:42:28.927510 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.927495    4528 scope.go:122] "RemoveContainer" containerID="9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d"
Nov 02 22:42:28.935496 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.935481    4528 scope.go:122] "RemoveContainer" containerID="b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b"
Nov 02 22:42:28.935728 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:42:28.935707    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b\": container with ID starting with b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b not found: ID does not exist" containerID="b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b"
Nov 02 22:42:28.935812 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.935734    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b"} err="failed to get container status \"b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b\": rpc error: code = NotFound desc = could not find container \"b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b\": container with ID starting with b9c946ce50d700fb7641e72ba874b7ad590cac20348f710340f54f66a5cefc7b not found: ID does not exist"
Nov 02 22:42:28.935812 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.935751    4528 scope.go:122] "RemoveContainer" containerID="9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d"
Nov 02 22:42:28.935934 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:42:28.935919    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d\": container with ID starting with 9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d not found: ID does not exist" containerID="9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d"
Nov 02 22:42:28.935975 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:28.935939    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d"} err="failed to get container status \"9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d\": rpc error: code = NotFound desc = could not find container \"9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d\": container with ID starting with 9ed9e81a1d2ad89fe91833828b1e8f0214aa93f9e4652387c1ed1de268167f3d not found: ID does not exist"
Nov 02 22:42:29.082636 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:29.082602    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"]
Nov 02 22:42:29.086241 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:29.086216    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6faf18-openshift-default-f9b76cdcc-v9zhr"]
Nov 02 22:42:30.912007 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:30.911976    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" path="/var/lib/kubelet/pods/2e6b3597-121b-4f4b-aab9-031a2124ff0f/volumes"
Nov 02 22:42:37.167074 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:37.167042    4528 scope.go:122] "RemoveContainer" containerID="605ecd7843768a88cede78befcfce7b7aa996315b50ee10ed9422cb42d497489"
Nov 02 22:42:43.782254 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:42:43.782214    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:42:44.111919 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:42:44.111830    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:43:07.416871 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:07.416836    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:43:10.870297 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:10.870249    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:43:10.870890 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:10.870247    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:43:10.870991 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:10.870961    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:43:10.871057 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:10.870862    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:43:14.111517 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:43:14.111483    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:43:25.903803 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.903767    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"]
Nov 02 22:43:25.904335 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904301    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d75bc0c3-a857-40fc-ba96-50483c3bf532" containerName="payload-pre-processing"
Nov 02 22:43:25.904335 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904321    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75bc0c3-a857-40fc-ba96-50483c3bf532" containerName="payload-pre-processing"
Nov 02 22:43:25.904469 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904349    4528 cpu_manager.go:414] "containerMap: removing container" podUID="455e8421-a549-498c-917e-361d5d3d7144" containerName="payload-processing"
Nov 02 22:43:25.904469 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904357    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455e8421-a549-498c-917e-361d5d3d7144" containerName="payload-processing"
Nov 02 22:43:25.904469 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904381    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy"
Nov 02 22:43:25.904469 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904388    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy"
Nov 02 22:43:25.904469 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904402    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy"
Nov 02 22:43:25.904469 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904410    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy"
Nov 02 22:43:25.904777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904512    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="455e8421-a549-498c-917e-361d5d3d7144" containerName="payload-processing"
Nov 02 22:43:25.904777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904526    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy"
Nov 02 22:43:25.904777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904534    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6b3597-121b-4f4b-aab9-031a2124ff0f" containerName="istio-proxy"
Nov 02 22:43:25.904777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.904548    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75bc0c3-a857-40fc-ba96-50483c3bf532" containerName="payload-pre-processing"
Nov 02 22:43:25.907708 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.907688    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:25.964203 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.964171    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"]
Nov 02 22:43:25.975167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.975143    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-tcb5m\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:25.975290 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.975188    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxkb\" (UniqueName: \"kubernetes.io/projected/c7cab708-1a60-4d0c-b0ef-11b6016facde-kube-api-access-wvxkb\") pod \"maas-api-revoke-sub-452b6d7b-tcb5m\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:25.975290 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:25.975208    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-tcb5m\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:26.012735 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.012705    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:43:26.012856 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.012740    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:43:26.012856 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.012754    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:43:26.012856 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.012772    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:43:26.076601 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.076575    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxkb\" (UniqueName: \"kubernetes.io/projected/c7cab708-1a60-4d0c-b0ef-11b6016facde-kube-api-access-wvxkb\") pod \"maas-api-revoke-sub-452b6d7b-tcb5m\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:26.076717 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.076611    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-tcb5m\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:26.076717 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.076693    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-tcb5m\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:26.077284 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.077265    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-tcb5m\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:26.078962 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.078939    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-tcb5m\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:26.133205 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.133180    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxkb\" (UniqueName: \"kubernetes.io/projected/c7cab708-1a60-4d0c-b0ef-11b6016facde-kube-api-access-wvxkb\") pod \"maas-api-revoke-sub-452b6d7b-tcb5m\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:26.221679 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.221622    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:26.584754 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:43:26.584721    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cab708_1a60_4d0c_b0ef_11b6016facde.slice/crio-837f072f46dbc98777ae543969b2d8f03c1bf74ae6faef591ff1e3173a0b0b33 WatchSource:0}: Error finding container 837f072f46dbc98777ae543969b2d8f03c1bf74ae6faef591ff1e3173a0b0b33: Status 404 returned error can't find the container with id 837f072f46dbc98777ae543969b2d8f03c1bf74ae6faef591ff1e3173a0b0b33
Nov 02 22:43:26.586723 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:26.586700    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"]
Nov 02 22:43:27.218053 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:27.217958    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m" event={"ID":"c7cab708-1a60-4d0c-b0ef-11b6016facde","Type":"ContainerStarted","Data":"a2a11605c9484a000841afad728948f4980ef7421e7c2b3dbac94f9c00d7d4a4"}
Nov 02 22:43:27.218053 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:27.218015    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m" event={"ID":"c7cab708-1a60-4d0c-b0ef-11b6016facde","Type":"ContainerStarted","Data":"837f072f46dbc98777ae543969b2d8f03c1bf74ae6faef591ff1e3173a0b0b33"}
Nov 02 22:43:27.331759 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:27.331710    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m" podStartSLOduration=2.331695794 podStartE2EDuration="2.331695794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:43:27.33000882 +0000 UTC m=+5694.593545137" watchObservedRunningTime="2026-11-02 22:43:27.331695794 +0000 UTC m=+5694.595232112"
Nov 02 22:43:27.479491 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:27.479465    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:43:28.223671 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:28.223635    4528 generic.go:356] "Generic (PLEG): container finished" podID="c7cab708-1a60-4d0c-b0ef-11b6016facde" containerID="a2a11605c9484a000841afad728948f4980ef7421e7c2b3dbac94f9c00d7d4a4" exitCode=0
Nov 02 22:43:28.224066 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:28.223677    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m" event={"ID":"c7cab708-1a60-4d0c-b0ef-11b6016facde","Type":"ContainerDied","Data":"a2a11605c9484a000841afad728948f4980ef7421e7c2b3dbac94f9c00d7d4a4"}
Nov 02 22:43:29.585126 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.585083    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:29.675029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.674266    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-openshift-ca\") pod \"c7cab708-1a60-4d0c-b0ef-11b6016facde\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") "
Nov 02 22:43:29.675029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.674345    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-xks-ca\") pod \"c7cab708-1a60-4d0c-b0ef-11b6016facde\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") "
Nov 02 22:43:29.675029 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.674448    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cab708-1a60-4d0c-b0ef-11b6016facde-kube-api-access-wvxkb\" (UniqueName: \"kubernetes.io/projected/c7cab708-1a60-4d0c-b0ef-11b6016facde-kube-api-access-wvxkb\") pod \"c7cab708-1a60-4d0c-b0ef-11b6016facde\" (UID: \"c7cab708-1a60-4d0c-b0ef-11b6016facde\") "
Nov 02 22:43:29.675806 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.675779    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-openshift-ca" pod "c7cab708-1a60-4d0c-b0ef-11b6016facde" (UID: "c7cab708-1a60-4d0c-b0ef-11b6016facde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:43:29.677352 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.677324    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cab708-1a60-4d0c-b0ef-11b6016facde-kube-api-access-wvxkb" pod "c7cab708-1a60-4d0c-b0ef-11b6016facde" (UID: "c7cab708-1a60-4d0c-b0ef-11b6016facde"). InnerVolumeSpecName "kube-api-access-wvxkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:43:29.678535 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.678501    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-xks-ca" pod "c7cab708-1a60-4d0c-b0ef-11b6016facde" (UID: "c7cab708-1a60-4d0c-b0ef-11b6016facde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:43:29.776073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.775968    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:43:29.776073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.776026    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cab708-1a60-4d0c-b0ef-11b6016facde-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:43:29.776073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:29.776044    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvxkb\" (UniqueName: \"kubernetes.io/projected/c7cab708-1a60-4d0c-b0ef-11b6016facde-kube-api-access-wvxkb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:43:30.276803 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:30.276767    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m" event={"ID":"c7cab708-1a60-4d0c-b0ef-11b6016facde","Type":"ContainerDied","Data":"837f072f46dbc98777ae543969b2d8f03c1bf74ae6faef591ff1e3173a0b0b33"}
Nov 02 22:43:30.277103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:30.277082    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="837f072f46dbc98777ae543969b2d8f03c1bf74ae6faef591ff1e3173a0b0b33"
Nov 02 22:43:30.277333 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:30.277317    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"
Nov 02 22:43:41.653667 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:43:41.653633    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:43:44.180437 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:43:44.180382    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:44:00.479485 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:44:00.479449    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:44:12.854922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:44:12.854887    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:44:14.190662 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:44:14.190609    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:44:24.166146 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:44:24.166089    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:44:38.773176 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:44:38.773138    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:44:43.044946 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:44:43.044914    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"]
Nov 02 22:44:43.095219 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:44:43.095181    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe07478-d8gw4"]
Nov 02 22:44:44.111538 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:44:44.111491    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:44:44.911032 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:44:44.910998    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4c2d48-c3d5-416d-a3cd-b46258c63ced" path="/var/lib/kubelet/pods/4f4c2d48-c3d5-416d-a3cd-b46258c63ced/volumes"
Nov 02 22:44:55.064391 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:44:55.064355    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"]
Nov 02 22:45:00.065357 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.065289    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6" podUID="60623367-8d1f-47b8-b90c-d7830b8a73b2" containerName="payload-pre-processing" containerID="cri-o://e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f" gracePeriod=25
Nov 02 22:45:00.322278 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.322221    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 22:45:00.341623 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.341598    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60623367-8d1f-47b8-b90c-d7830b8a73b2-kube-api-access-brt9p\" (UniqueName: \"kubernetes.io/projected/60623367-8d1f-47b8-b90c-d7830b8a73b2-kube-api-access-brt9p\") pod \"60623367-8d1f-47b8-b90c-d7830b8a73b2\" (UID: \"60623367-8d1f-47b8-b90c-d7830b8a73b2\") "
Nov 02 22:45:00.341743 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.341653    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60623367-8d1f-47b8-b90c-d7830b8a73b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60623367-8d1f-47b8-b90c-d7830b8a73b2-plugins-config-volume\") pod \"60623367-8d1f-47b8-b90c-d7830b8a73b2\" (UID: \"60623367-8d1f-47b8-b90c-d7830b8a73b2\") "
Nov 02 22:45:00.342102 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.342077    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60623367-8d1f-47b8-b90c-d7830b8a73b2-plugins-config-volume" pod "60623367-8d1f-47b8-b90c-d7830b8a73b2" (UID: "60623367-8d1f-47b8-b90c-d7830b8a73b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:45:00.343624 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.343604    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60623367-8d1f-47b8-b90c-d7830b8a73b2-kube-api-access-brt9p" pod "60623367-8d1f-47b8-b90c-d7830b8a73b2" (UID: "60623367-8d1f-47b8-b90c-d7830b8a73b2"). InnerVolumeSpecName "kube-api-access-brt9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:45:00.442718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.442685    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60623367-8d1f-47b8-b90c-d7830b8a73b2-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:45:00.442718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.442719    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brt9p\" (UniqueName: \"kubernetes.io/projected/60623367-8d1f-47b8-b90c-d7830b8a73b2-kube-api-access-brt9p\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:45:00.719906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.719825    4528 generic.go:356] "Generic (PLEG): container finished" podID="60623367-8d1f-47b8-b90c-d7830b8a73b2" containerID="e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f" exitCode=0
Nov 02 22:45:00.719906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.719883    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"
Nov 02 22:45:00.720096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.719918    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6" event={"ID":"60623367-8d1f-47b8-b90c-d7830b8a73b2","Type":"ContainerDied","Data":"e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f"}
Nov 02 22:45:00.720096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.719991    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6" event={"ID":"60623367-8d1f-47b8-b90c-d7830b8a73b2","Type":"ContainerDied","Data":"324a5c31039320466de741f9cd07c3d36ed3f9e7cd1903a45932fa97d2e69a70"}
Nov 02 22:45:00.720096 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.720012    4528 scope.go:122] "RemoveContainer" containerID="e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f"
Nov 02 22:45:00.728502 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.728484    4528 scope.go:122] "RemoveContainer" containerID="e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f"
Nov 02 22:45:00.728768 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:45:00.728748    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f\": container with ID starting with e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f not found: ID does not exist" containerID="e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f"
Nov 02 22:45:00.728847 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.728775    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f"} err="failed to get container status \"e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f\": rpc error: code = NotFound desc = could not find container \"e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f\": container with ID starting with e52703f71c7a2d582e2b4afddb08566f0014d6e1767112505ee21c6dfa5b723f not found: ID does not exist"
Nov 02 22:45:00.882827 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.882794    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"]
Nov 02 22:45:00.932785 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:00.932758    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kzlc6"]
Nov 02 22:45:01.045976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:01.045946    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"]
Nov 02 22:45:01.048514 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:01.048492    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08e6842-gqj7s"]
Nov 02 22:45:02.911482 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:02.911447    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbf0d00-8b7b-4f6a-992e-684686f0db07" path="/var/lib/kubelet/pods/2bbf0d00-8b7b-4f6a-992e-684686f0db07/volumes"
Nov 02 22:45:02.911865 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:02.911805    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60623367-8d1f-47b8-b90c-d7830b8a73b2" path="/var/lib/kubelet/pods/60623367-8d1f-47b8-b90c-d7830b8a73b2/volumes"
Nov 02 22:45:09.138577 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:09.138544    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"]
Nov 02 22:45:11.222731 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.222698    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6cd6f7444d-c7gdn"]
Nov 02 22:45:11.223247 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.223058    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c7cab708-1a60-4d0c-b0ef-11b6016facde" containerName="revoke-keys"
Nov 02 22:45:11.223247 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.223070    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7cab708-1a60-4d0c-b0ef-11b6016facde" containerName="revoke-keys"
Nov 02 22:45:11.223247 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.223093    4528 cpu_manager.go:414] "containerMap: removing container" podUID="60623367-8d1f-47b8-b90c-d7830b8a73b2" containerName="payload-pre-processing"
Nov 02 22:45:11.223247 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.223101    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60623367-8d1f-47b8-b90c-d7830b8a73b2" containerName="payload-pre-processing"
Nov 02 22:45:11.223247 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.223197    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7cab708-1a60-4d0c-b0ef-11b6016facde" containerName="revoke-keys"
Nov 02 22:45:11.223247 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.223207    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="60623367-8d1f-47b8-b90c-d7830b8a73b2" containerName="payload-pre-processing"
Nov 02 22:45:11.227536 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.227518    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:11.283352 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.283330    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6cd6f7444d-c7gdn"]
Nov 02 22:45:11.335449 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.335417    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c74zd\" (UniqueName: \"kubernetes.io/projected/d0d9de36-b748-4d05-9e97-2182dbed6f6b-kube-api-access-c74zd\") pod \"payload-processing-6cd6f7444d-c7gdn\" (UID: \"d0d9de36-b748-4d05-9e97-2182dbed6f6b\") " pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:11.335626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.335552    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d9de36-b748-4d05-9e97-2182dbed6f6b-plugins-config-volume\") pod \"payload-processing-6cd6f7444d-c7gdn\" (UID: \"d0d9de36-b748-4d05-9e97-2182dbed6f6b\") " pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:11.374454 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.374424    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"]
Nov 02 22:45:11.378293 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.378277    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:11.434278 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.434253    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"]
Nov 02 22:45:11.436730 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.436708    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1bf636-d125-4f0c-b76f-6a11c1f26457-plugins-config-volume\") pod \"payload-pre-processing-7669ccf7ff-28hfp\" (UID: \"cc1bf636-d125-4f0c-b76f-6a11c1f26457\") " pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:11.436844 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.436740    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d9de36-b748-4d05-9e97-2182dbed6f6b-plugins-config-volume\") pod \"payload-processing-6cd6f7444d-c7gdn\" (UID: \"d0d9de36-b748-4d05-9e97-2182dbed6f6b\") " pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:11.436915 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.436832    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pft5\" (UniqueName: \"kubernetes.io/projected/cc1bf636-d125-4f0c-b76f-6a11c1f26457-kube-api-access-5pft5\") pod \"payload-pre-processing-7669ccf7ff-28hfp\" (UID: \"cc1bf636-d125-4f0c-b76f-6a11c1f26457\") " pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:11.436915 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.436884    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c74zd\" (UniqueName: \"kubernetes.io/projected/d0d9de36-b748-4d05-9e97-2182dbed6f6b-kube-api-access-c74zd\") pod \"payload-processing-6cd6f7444d-c7gdn\" (UID: \"d0d9de36-b748-4d05-9e97-2182dbed6f6b\") " pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:11.437528 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.437501    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d9de36-b748-4d05-9e97-2182dbed6f6b-plugins-config-volume\") pod \"payload-processing-6cd6f7444d-c7gdn\" (UID: \"d0d9de36-b748-4d05-9e97-2182dbed6f6b\") " pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:11.496922 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.496857    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c74zd\" (UniqueName: \"kubernetes.io/projected/d0d9de36-b748-4d05-9e97-2182dbed6f6b-kube-api-access-c74zd\") pod \"payload-processing-6cd6f7444d-c7gdn\" (UID: \"d0d9de36-b748-4d05-9e97-2182dbed6f6b\") " pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:11.537451 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.537422    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pft5\" (UniqueName: \"kubernetes.io/projected/cc1bf636-d125-4f0c-b76f-6a11c1f26457-kube-api-access-5pft5\") pod \"payload-pre-processing-7669ccf7ff-28hfp\" (UID: \"cc1bf636-d125-4f0c-b76f-6a11c1f26457\") " pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:11.537685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.537527    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1bf636-d125-4f0c-b76f-6a11c1f26457-plugins-config-volume\") pod \"payload-pre-processing-7669ccf7ff-28hfp\" (UID: \"cc1bf636-d125-4f0c-b76f-6a11c1f26457\") " pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:11.538401 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.538376    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1bf636-d125-4f0c-b76f-6a11c1f26457-plugins-config-volume\") pod \"payload-pre-processing-7669ccf7ff-28hfp\" (UID: \"cc1bf636-d125-4f0c-b76f-6a11c1f26457\") " pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:11.539236 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.539217    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:11.594648 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.594584    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pft5\" (UniqueName: \"kubernetes.io/projected/cc1bf636-d125-4f0c-b76f-6a11c1f26457-kube-api-access-5pft5\") pod \"payload-pre-processing-7669ccf7ff-28hfp\" (UID: \"cc1bf636-d125-4f0c-b76f-6a11c1f26457\") " pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:11.690226 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.690200    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:11.903864 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:45:11.903829    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d9de36_b748_4d05_9e97_2182dbed6f6b.slice/crio-fefc0b4e523c8ad9171bfc9d2503e22204879a87df9eadab77cb80a9bdbadcf9 WatchSource:0}: Error finding container fefc0b4e523c8ad9171bfc9d2503e22204879a87df9eadab77cb80a9bdbadcf9: Status 404 returned error can't find the container with id fefc0b4e523c8ad9171bfc9d2503e22204879a87df9eadab77cb80a9bdbadcf9
Nov 02 22:45:11.905342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:11.905319    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6cd6f7444d-c7gdn"]
Nov 02 22:45:12.055228 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:45:12.055195    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1bf636_d125_4f0c_b76f_6a11c1f26457.slice/crio-2f2ba370b77dda527f93ea3ad695b4814fb9e7103d8825539e531de0e22f5542 WatchSource:0}: Error finding container 2f2ba370b77dda527f93ea3ad695b4814fb9e7103d8825539e531de0e22f5542: Status 404 returned error can't find the container with id 2f2ba370b77dda527f93ea3ad695b4814fb9e7103d8825539e531de0e22f5542
Nov 02 22:45:12.055360 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:12.055233    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"]
Nov 02 22:45:12.777300 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:12.777267    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn" event={"ID":"d0d9de36-b748-4d05-9e97-2182dbed6f6b","Type":"ContainerStarted","Data":"286f661c2626d1b4d8961dbc2a6b546b5be4388d95a7755d344a1b173088cd03"}
Nov 02 22:45:12.777780 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:12.777319    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn" event={"ID":"d0d9de36-b748-4d05-9e97-2182dbed6f6b","Type":"ContainerStarted","Data":"fefc0b4e523c8ad9171bfc9d2503e22204879a87df9eadab77cb80a9bdbadcf9"}
Nov 02 22:45:12.777780 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:12.777374    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:12.778933 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:12.778906    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp" event={"ID":"cc1bf636-d125-4f0c-b76f-6a11c1f26457","Type":"ContainerStarted","Data":"26ac92fe8d31c19b8f6e95bed26f691ce33baf5722bd067c04629b82453822aa"}
Nov 02 22:45:12.779058 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:12.778942    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp" event={"ID":"cc1bf636-d125-4f0c-b76f-6a11c1f26457","Type":"ContainerStarted","Data":"2f2ba370b77dda527f93ea3ad695b4814fb9e7103d8825539e531de0e22f5542"}
Nov 02 22:45:12.779058 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:12.778978    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:12.894821 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:12.894765    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn" podStartSLOduration=1.8947468280000002 podStartE2EDuration="1.894746828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:45:12.892331607 +0000 UTC m=+5800.155867923" watchObservedRunningTime="2026-11-02 22:45:12.894746828 +0000 UTC m=+5800.158283146"
Nov 02 22:45:13.006592 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:13.006536    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp" podStartSLOduration=2.006519652 podStartE2EDuration="2.006519652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:45:13.005219818 +0000 UTC m=+5800.268756135" watchObservedRunningTime="2026-11-02 22:45:13.006519652 +0000 UTC m=+5800.270055969"
Nov 02 22:45:14.112343 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:45:14.112308    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:45:14.139079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.139039    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv" podUID="53e705a3-6036-4fdd-800c-90b4eb36f5a8" containerName="payload-processing" containerID="cri-o://4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99" gracePeriod=25
Nov 02 22:45:14.388130 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.388078    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 22:45:14.466183 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.466155    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e705a3-6036-4fdd-800c-90b4eb36f5a8-kube-api-access-kn8kh\" (UniqueName: \"kubernetes.io/projected/53e705a3-6036-4fdd-800c-90b4eb36f5a8-kube-api-access-kn8kh\") pod \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\" (UID: \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\") "
Nov 02 22:45:14.466337 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.466193    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e705a3-6036-4fdd-800c-90b4eb36f5a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e705a3-6036-4fdd-800c-90b4eb36f5a8-plugins-config-volume\") pod \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\" (UID: \"53e705a3-6036-4fdd-800c-90b4eb36f5a8\") "
Nov 02 22:45:14.466613 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.466593    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e705a3-6036-4fdd-800c-90b4eb36f5a8-plugins-config-volume" pod "53e705a3-6036-4fdd-800c-90b4eb36f5a8" (UID: "53e705a3-6036-4fdd-800c-90b4eb36f5a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:45:14.468191 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.468164    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e705a3-6036-4fdd-800c-90b4eb36f5a8-kube-api-access-kn8kh" pod "53e705a3-6036-4fdd-800c-90b4eb36f5a8" (UID: "53e705a3-6036-4fdd-800c-90b4eb36f5a8"). InnerVolumeSpecName "kube-api-access-kn8kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:45:14.567630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.567604    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8kh\" (UniqueName: \"kubernetes.io/projected/53e705a3-6036-4fdd-800c-90b4eb36f5a8-kube-api-access-kn8kh\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:45:14.567630 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.567627    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e705a3-6036-4fdd-800c-90b4eb36f5a8-plugins-config-volume\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:45:14.788916 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.788879    4528 generic.go:356] "Generic (PLEG): container finished" podID="53e705a3-6036-4fdd-800c-90b4eb36f5a8" containerID="4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99" exitCode=0
Nov 02 22:45:14.789167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.788926    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"
Nov 02 22:45:14.789167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.788961    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv" event={"ID":"53e705a3-6036-4fdd-800c-90b4eb36f5a8","Type":"ContainerDied","Data":"4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99"}
Nov 02 22:45:14.789167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.789026    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pwkmv" event={"ID":"53e705a3-6036-4fdd-800c-90b4eb36f5a8","Type":"ContainerDied","Data":"f01ecfaaeda10d8a2f803f564a05b051848c84cc55908d2703c416810a383c0f"}
Nov 02 22:45:14.789167 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.789055    4528 scope.go:122] "RemoveContainer" containerID="4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99"
Nov 02 22:45:14.798511 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.798495    4528 scope.go:122] "RemoveContainer" containerID="4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99"
Nov 02 22:45:14.798763 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:45:14.798742    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99\": container with ID starting with 4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99 not found: ID does not exist" containerID="4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99"
Nov 02 22:45:14.798813 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.798771    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99"} err="failed to get container status \"4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99\": rpc error: code = NotFound desc = could not find container \"4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99\": container with ID starting with 4d4896a726d5f0e469a02efe878941eb8dec38c78e79b8339d47574ec500ba99 not found: ID does not exist"
Nov 02 22:45:14.953194 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.953169    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"]
Nov 02 22:45:14.956776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:14.956756    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pwkmv"]
Nov 02 22:45:16.917906 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:16.917873    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e705a3-6036-4fdd-800c-90b4eb36f5a8" path="/var/lib/kubelet/pods/53e705a3-6036-4fdd-800c-90b4eb36f5a8/volumes"
Nov 02 22:45:23.786004 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:23.785975    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6cd6f7444d-c7gdn"
Nov 02 22:45:23.786412 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:23.786043    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7669ccf7ff-28hfp"
Nov 02 22:45:31.349076 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:45:31.349043    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:45:44.112326 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:45:44.112286    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:46:14.111581 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:46:14.111546    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:46:37.224973 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:46:37.224939    4528 scope.go:122] "RemoveContainer" containerID="89bad85f1585f7d7ce070c6e2da827145626c200937c53196b6fcce8d4250c20"
Nov 02 22:46:37.232889 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:46:37.232787    4528 scope.go:122] "RemoveContainer" containerID="88ee0b6a6658ca737b2c1e264aeb314620b7f7e62c335d49019be33c49032254"
Nov 02 22:46:44.111802 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:46:44.111767    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:47:14.112333 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:47:14.112290    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:47:18.275187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:47:18.275153    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:47:28.173171 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:47:28.173133    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:47:36.672703 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:47:36.672661    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:47:44.111878 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:47:44.111838    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:47:51.760103 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:47:51.760069    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:48:02.885823 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:48:02.885790    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:48:14.111703 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:48:14.111656    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:48:20.970718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:48:20.970685    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:48:30.044902 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:48:30.044858    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"]
Nov 02 22:48:30.095287 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:48:30.095052    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tcb5m"]
Nov 02 22:48:30.911866 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:48:30.911835    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7cab708-1a60-4d0c-b0ef-11b6016facde" path="/var/lib/kubelet/pods/c7cab708-1a60-4d0c-b0ef-11b6016facde/volumes"
Nov 02 22:48:34.072615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:48:34.072579    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:48:44.115627 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:48:44.115582    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:49:04.574453 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:49:04.574414    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:49:14.112187 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:49:14.112150    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:49:15.272453 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:49:15.272421    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:49:26.446018 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:49:26.445983    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:49:37.268030 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:49:37.268003    4528 scope.go:122] "RemoveContainer" containerID="a2a11605c9484a000841afad728948f4980ef7421e7c2b3dbac94f9c00d7d4a4"
Nov 02 22:49:44.112913 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:49:44.112867    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:49:49.790223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:49:49.790187    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:49:59.663620 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:49:59.663585    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:50:08.474856 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:50:08.474820    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:50:14.112459 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:50:14.112414    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:50:17.880266 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:50:17.880228    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:50:28.740697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:50:28.740661    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:50:41.546201 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:50:41.546163    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:50:44.112088 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:50:44.112052    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:50:50.253957 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:50:50.253926    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:51:03.045932 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:51:03.045898    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:51:13.975843 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:51:13.975809    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:51:14.112462 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:51:14.112427    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:51:29.348492 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:51:29.348414    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:51:39.645342 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:51:39.645306    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:51:44.112219 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:51:44.112184    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:51:51.533301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:51:51.533267    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:52:12.580907 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:12.580870    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:52:14.179297 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:52:14.179254    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:52:23.259217 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:23.259177    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:52:42.042867 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:42.042833    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:52:44.146186 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:52:44.146126    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:52:48.667520 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.667487    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"]
Nov 02 22:52:48.667963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.667901    4528 cpu_manager.go:414] "containerMap: removing container" podUID="53e705a3-6036-4fdd-800c-90b4eb36f5a8" containerName="payload-processing"
Nov 02 22:52:48.667963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.667916    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e705a3-6036-4fdd-800c-90b4eb36f5a8" containerName="payload-processing"
Nov 02 22:52:48.668049 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.667990    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e705a3-6036-4fdd-800c-90b4eb36f5a8" containerName="payload-processing"
Nov 02 22:52:48.671090 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.671075    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.737237 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.737199    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-credential-socket\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.737463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.737251    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-certs\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.737463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.737283    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-socket\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.737463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.737331    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-data\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.737463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.737381    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d496f-0554-4834-9f38-70d09665112e-istio-podinfo\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.737463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.737436    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-envoy\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.737782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.737468    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-istio-token\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.737782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.737492    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d496f-0554-4834-9f38-70d09665112e-istiod-ca-cert\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.737782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.737523    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7d2w\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-kube-api-access-b7d2w\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.747313 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.747289    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-90aa9a-openshift-default-dockercfg-c9wss\""
Nov 02 22:52:48.776259 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.776230    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"]
Nov 02 22:52:48.838480 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.838440    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d496f-0554-4834-9f38-70d09665112e-istio-podinfo\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.838682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.838524    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-envoy\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.838682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.838557    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-istio-token\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.838682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.838579    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d496f-0554-4834-9f38-70d09665112e-istiod-ca-cert\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.838682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.838610    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7d2w\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-kube-api-access-b7d2w\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.838682 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.838668    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-credential-socket\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.838964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.838699    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-certs\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.838964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.838728    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-socket\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.838964 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.838781    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-data\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.839507 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.839417    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-certs\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.839507 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.839468    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-data\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.839717 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.839695    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-credential-socket\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.839803 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.839769    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d496f-0554-4834-9f38-70d09665112e-istiod-ca-cert\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.839860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.839796    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-socket\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.842187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.842163    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-envoy\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.842304 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.842244    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d496f-0554-4834-9f38-70d09665112e-istio-podinfo\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.897526 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.897494    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-istio-token\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.897870 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.897852    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7d2w\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-kube-api-access-b7d2w\") pod \"e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") " pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:48.984693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:48.983786    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:49.407414 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:52:49.406651    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8d496f_0554_4834_9f38_70d09665112e.slice/crio-4f8a424cbe142b293f9e90d9a25294a8d70560a14d4d224c4e7ce529064a3dc6 WatchSource:0}: Error finding container 4f8a424cbe142b293f9e90d9a25294a8d70560a14d4d224c4e7ce529064a3dc6: Status 404 returned error can't find the container with id 4f8a424cbe142b293f9e90d9a25294a8d70560a14d4d224c4e7ce529064a3dc6
Nov 02 22:52:49.408329 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:49.408295    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"]
Nov 02 22:52:49.408989 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:49.408958    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:52:49.409105 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:49.409032    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:52:49.409105 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:49.409076    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:52:49.958445 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:49.958411    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" event={"ID":"8b8d496f-0554-4834-9f38-70d09665112e","Type":"ContainerStarted","Data":"435ebb59ab21e539390ee1be9c9c15e3f3b1a1e89319b80ba0b174c72bf935bb"}
Nov 02 22:52:49.958836 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:49.958463    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" event={"ID":"8b8d496f-0554-4834-9f38-70d09665112e","Type":"ContainerStarted","Data":"4f8a424cbe142b293f9e90d9a25294a8d70560a14d4d224c4e7ce529064a3dc6"}
Nov 02 22:52:49.984772 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:49.984739    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:50.090767 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:50.090714    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" podStartSLOduration=2.090700687 podStartE2EDuration="2.090700687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:50.088750242 +0000 UTC m=+6257.352286555" watchObservedRunningTime="2026-11-02 22:52:50.090700687 +0000 UTC m=+6257.354237004"
Nov 02 22:52:50.989614 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:50.989584    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:51.968367 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:51.968338    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:51.969307 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:51.969290    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:52:53.566186 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:52:53.566150    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:53:07.670529 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:07.670493    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:53:14.112630 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:53:14.112584    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:53:16.087547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:16.087519    4528 generic.go:356] "Generic (PLEG): container finished" podID="8b8d496f-0554-4834-9f38-70d09665112e" containerID="435ebb59ab21e539390ee1be9c9c15e3f3b1a1e89319b80ba0b174c72bf935bb" exitCode=0
Nov 02 22:53:16.087904 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:16.087591    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" event={"ID":"8b8d496f-0554-4834-9f38-70d09665112e","Type":"ContainerDied","Data":"435ebb59ab21e539390ee1be9c9c15e3f3b1a1e89319b80ba0b174c72bf935bb"}
Nov 02 22:53:16.087963 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:16.087941    4528 scope.go:122] "RemoveContainer" containerID="435ebb59ab21e539390ee1be9c9c15e3f3b1a1e89319b80ba0b174c72bf935bb"
Nov 02 22:53:16.089041 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:16.089010    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:53:16.089127 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:16.089074    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:53:16.089127 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:16.089105    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 22:53:17.094305 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:17.094272    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" event={"ID":"8b8d496f-0554-4834-9f38-70d09665112e","Type":"ContainerStarted","Data":"304c3bf864c08e12c7785b374106a3a0ff8fc7fa3d387a7377aeecbaedfaa832"}
Nov 02 22:53:17.984742 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:17.984702    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:53:17.986152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:17.986063    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body=
Nov 02 22:53:17.986152 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:17.986142    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused"
Nov 02 22:53:18.984860 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:18.984826    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body=
Nov 02 22:53:18.985412 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:18.984884    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused"
Nov 02 22:53:19.985147 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:19.985089    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body=
Nov 02 22:53:19.985545 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:19.985165    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused"
Nov 02 22:53:20.985154 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:20.985099    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body=
Nov 02 22:53:20.985567 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:20.985175    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused"
Nov 02 22:53:21.968748 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:21.968708    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:53:22.012413 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:22.012380    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:53:22.138384 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:22.138355    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:53:25.550923 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:25.550885    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:53:44.112234 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:53:44.112198    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:53:53.482314 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:53:53.482279    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:54:07.637182 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.637152    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"]
Nov 02 22:54:07.640679 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.640657    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.707296 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.707261    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mh5\" (UniqueName: \"kubernetes.io/projected/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-kube-api-access-r4mh5\") pod \"maas-api-revoke-sub-91c04135-j6wvm\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.707450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.707377    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91c04135-j6wvm\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.707450 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.707439    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91c04135-j6wvm\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.745654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.745615    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:54:07.745654 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.745618    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:54:07.745849 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.745674    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:54:07.745849 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.745695    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:54:07.747328 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.747308    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"]
Nov 02 22:54:07.808632 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.808610    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91c04135-j6wvm\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.808797 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.808665    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91c04135-j6wvm\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.808797 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.808715    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mh5\" (UniqueName: \"kubernetes.io/projected/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-kube-api-access-r4mh5\") pod \"maas-api-revoke-sub-91c04135-j6wvm\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.809362 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.809339    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91c04135-j6wvm\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.810928 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.810911    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91c04135-j6wvm\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.864875 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.864850    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mh5\" (UniqueName: \"kubernetes.io/projected/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-kube-api-access-r4mh5\") pod \"maas-api-revoke-sub-91c04135-j6wvm\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:07.953181 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:07.953127    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:08.316793 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:54:08.316754    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a7c4da_1fa7_4f75_8298_f857cdf5cddb.slice/crio-c33b1834893745fd8c528643d5609aafc987ccd5d2222c25c8e1b8af683f240a WatchSource:0}: Error finding container c33b1834893745fd8c528643d5609aafc987ccd5d2222c25c8e1b8af683f240a: Status 404 returned error can't find the container with id c33b1834893745fd8c528643d5609aafc987ccd5d2222c25c8e1b8af683f240a
Nov 02 22:54:08.318466 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:08.318446    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"]
Nov 02 22:54:08.350434 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:08.350409    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm" event={"ID":"31a7c4da-1fa7-4f75-8298-f857cdf5cddb","Type":"ContainerStarted","Data":"c33b1834893745fd8c528643d5609aafc987ccd5d2222c25c8e1b8af683f240a"}
Nov 02 22:54:09.356718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:09.356628    4528 generic.go:356] "Generic (PLEG): container finished" podID="31a7c4da-1fa7-4f75-8298-f857cdf5cddb" containerID="3ca996a1243403b6b00a92e6ef14a146b11a753cbb11796e5efaa9175b363e07" exitCode=0
Nov 02 22:54:09.356718 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:09.356700    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm" event={"ID":"31a7c4da-1fa7-4f75-8298-f857cdf5cddb","Type":"ContainerDied","Data":"3ca996a1243403b6b00a92e6ef14a146b11a753cbb11796e5efaa9175b363e07"}
Nov 02 22:54:10.488696 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.488670    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:10.537093 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.537066    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-kube-api-access-r4mh5\" (UniqueName: \"kubernetes.io/projected/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-kube-api-access-r4mh5\") pod \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") "
Nov 02 22:54:10.537256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.537174    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-openshift-ca\") pod \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") "
Nov 02 22:54:10.537256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.537195    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-xks-ca\") pod \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\" (UID: \"31a7c4da-1fa7-4f75-8298-f857cdf5cddb\") "
Nov 02 22:54:10.537606 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.537578    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-openshift-ca" pod "31a7c4da-1fa7-4f75-8298-f857cdf5cddb" (UID: "31a7c4da-1fa7-4f75-8298-f857cdf5cddb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:54:10.538967 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.538948    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-kube-api-access-r4mh5" pod "31a7c4da-1fa7-4f75-8298-f857cdf5cddb" (UID: "31a7c4da-1fa7-4f75-8298-f857cdf5cddb"). InnerVolumeSpecName "kube-api-access-r4mh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:54:10.539149 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.539128    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-xks-ca" pod "31a7c4da-1fa7-4f75-8298-f857cdf5cddb" (UID: "31a7c4da-1fa7-4f75-8298-f857cdf5cddb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:54:10.638397 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.638318    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4mh5\" (UniqueName: \"kubernetes.io/projected/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-kube-api-access-r4mh5\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:10.638397 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.638357    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:10.638397 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:10.638373    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a7c4da-1fa7-4f75-8298-f857cdf5cddb-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:11.366363 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:11.366329    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm" event={"ID":"31a7c4da-1fa7-4f75-8298-f857cdf5cddb","Type":"ContainerDied","Data":"c33b1834893745fd8c528643d5609aafc987ccd5d2222c25c8e1b8af683f240a"}
Nov 02 22:54:11.366547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:11.366387    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c33b1834893745fd8c528643d5609aafc987ccd5d2222c25c8e1b8af683f240a"
Nov 02 22:54:11.366547 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:11.366337    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"
Nov 02 22:54:14.112042 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:54:14.112006    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:54:22.579761 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:22.579730    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:54:36.756059 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:54:36.756015    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:54:44.112716 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:54:44.112678    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:55:04.491854 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:04.491818    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"]
Nov 02 22:55:04.492448 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:04.492067    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy" containerID="cri-o://304c3bf864c08e12c7785b374106a3a0ff8fc7fa3d387a7377aeecbaedfaa832" gracePeriod=30
Nov 02 22:55:07.140345 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:07.140310    4528 patch_prober.go:29] interesting pod/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:55:07.140747 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:07.140366    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:55:09.620061 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.620028    4528 generic.go:356] "Generic (PLEG): container finished" podID="8b8d496f-0554-4834-9f38-70d09665112e" containerID="304c3bf864c08e12c7785b374106a3a0ff8fc7fa3d387a7377aeecbaedfaa832" exitCode=0
Nov 02 22:55:09.620481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.620096    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" event={"ID":"8b8d496f-0554-4834-9f38-70d09665112e","Type":"ContainerDied","Data":"304c3bf864c08e12c7785b374106a3a0ff8fc7fa3d387a7377aeecbaedfaa832"}
Nov 02 22:55:09.620481 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.620183    4528 scope.go:122] "RemoveContainer" containerID="435ebb59ab21e539390ee1be9c9c15e3f3b1a1e89319b80ba0b174c72bf935bb"
Nov 02 22:55:09.745068 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.745043    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:55:09.903765 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.903735    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-certs\") pod \"8b8d496f-0554-4834-9f38-70d09665112e\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") "
Nov 02 22:55:09.903961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.903781    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-envoy\") pod \"8b8d496f-0554-4834-9f38-70d09665112e\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") "
Nov 02 22:55:09.903961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.903822    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-kube-api-access-b7d2w\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-kube-api-access-b7d2w\") pod \"8b8d496f-0554-4834-9f38-70d09665112e\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") "
Nov 02 22:55:09.903961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.903866    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-data\") pod \"8b8d496f-0554-4834-9f38-70d09665112e\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") "
Nov 02 22:55:09.903961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.903914    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-istio-token\") pod \"8b8d496f-0554-4834-9f38-70d09665112e\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") "
Nov 02 22:55:09.903961 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.903943    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-credential-socket\") pod \"8b8d496f-0554-4834-9f38-70d09665112e\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") "
Nov 02 22:55:09.904275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.903977    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b8d496f-0554-4834-9f38-70d09665112e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d496f-0554-4834-9f38-70d09665112e-istio-podinfo\") pod \"8b8d496f-0554-4834-9f38-70d09665112e\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") "
Nov 02 22:55:09.904275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904017    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-certs" pod "8b8d496f-0554-4834-9f38-70d09665112e" (UID: "8b8d496f-0554-4834-9f38-70d09665112e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:55:09.904275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904050    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-socket\") pod \"8b8d496f-0554-4834-9f38-70d09665112e\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") "
Nov 02 22:55:09.904275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904099    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8d496f-0554-4834-9f38-70d09665112e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d496f-0554-4834-9f38-70d09665112e-istiod-ca-cert\") pod \"8b8d496f-0554-4834-9f38-70d09665112e\" (UID: \"8b8d496f-0554-4834-9f38-70d09665112e\") "
Nov 02 22:55:09.904275 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904264    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-credential-socket" pod "8b8d496f-0554-4834-9f38-70d09665112e" (UID: "8b8d496f-0554-4834-9f38-70d09665112e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:55:09.904552 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904277    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-data" pod "8b8d496f-0554-4834-9f38-70d09665112e" (UID: "8b8d496f-0554-4834-9f38-70d09665112e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:55:09.904552 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904484    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-socket" pod "8b8d496f-0554-4834-9f38-70d09665112e" (UID: "8b8d496f-0554-4834-9f38-70d09665112e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:55:09.904552 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904516    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-certs\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:09.904552 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904537    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-data\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:09.904729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904555    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-credential-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:09.904729 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.904591    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8d496f-0554-4834-9f38-70d09665112e-istiod-ca-cert" pod "8b8d496f-0554-4834-9f38-70d09665112e" (UID: "8b8d496f-0554-4834-9f38-70d09665112e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:55:09.906094 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.906070    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-kube-api-access-b7d2w" pod "8b8d496f-0554-4834-9f38-70d09665112e" (UID: "8b8d496f-0554-4834-9f38-70d09665112e"). InnerVolumeSpecName "kube-api-access-b7d2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:55:09.906278 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.906256    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b8d496f-0554-4834-9f38-70d09665112e-istio-podinfo" pod "8b8d496f-0554-4834-9f38-70d09665112e" (UID: "8b8d496f-0554-4834-9f38-70d09665112e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 22:55:09.906365 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.906348    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-envoy" pod "8b8d496f-0554-4834-9f38-70d09665112e" (UID: "8b8d496f-0554-4834-9f38-70d09665112e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:55:09.906472 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:09.906450    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-istio-token" pod "8b8d496f-0554-4834-9f38-70d09665112e" (UID: "8b8d496f-0554-4834-9f38-70d09665112e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:55:10.005534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.005504    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8d496f-0554-4834-9f38-70d09665112e-istio-podinfo\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:10.005534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.005530    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-workload-socket\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:10.005534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.005539    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8d496f-0554-4834-9f38-70d09665112e-istiod-ca-cert\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:10.005534 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.005549    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8d496f-0554-4834-9f38-70d09665112e-istio-envoy\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:10.005818 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.005558    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7d2w\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-kube-api-access-b7d2w\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:10.005818 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.005568    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8d496f-0554-4834-9f38-70d09665112e-istio-token\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:10.625756 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.625724    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs" event={"ID":"8b8d496f-0554-4834-9f38-70d09665112e","Type":"ContainerDied","Data":"4f8a424cbe142b293f9e90d9a25294a8d70560a14d4d224c4e7ce529064a3dc6"}
Nov 02 22:55:10.626197 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.625757    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"
Nov 02 22:55:10.626197 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.625778    4528 scope.go:122] "RemoveContainer" containerID="304c3bf864c08e12c7785b374106a3a0ff8fc7fa3d387a7377aeecbaedfaa832"
Nov 02 22:55:10.839984 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.839954    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"]
Nov 02 22:55:10.890343 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.890276    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-90aa9a-openshift-default-78bffd8cf7bfjbs"]
Nov 02 22:55:10.911848 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:10.911820    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8d496f-0554-4834-9f38-70d09665112e" path="/var/lib/kubelet/pods/8b8d496f-0554-4834-9f38-70d09665112e/volumes"
Nov 02 22:55:14.111554 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:55:14.111517    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:55:20.860344 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:55:20.860310    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:55:44.111579 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:55:44.111532    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:56:14.112805 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:56:14.112764    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:56:44.112821 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:56:44.112786    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:56:54.883553 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:56:54.883519    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:57:07.235280 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:07.235248    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:57:14.178342 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:57:14.178279    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:57:14.937615 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:14.937579    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:57:23.089940 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:23.089905    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:57:40.951084 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:40.951053    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:57:44.111798 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:57:44.111756    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:57:54.172392 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.172359    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"]
Nov 02 22:57:54.172834 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.172732    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy"
Nov 02 22:57:54.172834 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.172746    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy"
Nov 02 22:57:54.172834 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.172770    4528 cpu_manager.go:414] "containerMap: removing container" podUID="31a7c4da-1fa7-4f75-8298-f857cdf5cddb" containerName="revoke-keys"
Nov 02 22:57:54.172834 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.172776    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a7c4da-1fa7-4f75-8298-f857cdf5cddb" containerName="revoke-keys"
Nov 02 22:57:54.172988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.172853    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a7c4da-1fa7-4f75-8298-f857cdf5cddb" containerName="revoke-keys"
Nov 02 22:57:54.172988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.172866    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy"
Nov 02 22:57:54.172988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.172871    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy"
Nov 02 22:57:54.176003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.175986    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.208414 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.208381    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6wcg\" (UniqueName: \"kubernetes.io/projected/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-kube-api-access-p6wcg\") pod \"maas-api-revoke-sub-7cd3b896-bwjhx\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.208553 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.208424    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cd3b896-bwjhx\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.208625 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.208566    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cd3b896-bwjhx\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.280683 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.280651    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:57:54.280853 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.280709    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:57:54.280853 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.280710    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:57:54.280853 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.280786    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:57:54.282541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.282516    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"]
Nov 02 22:57:54.309951 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.309926    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cd3b896-bwjhx\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.310061 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.309967    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6wcg\" (UniqueName: \"kubernetes.io/projected/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-kube-api-access-p6wcg\") pod \"maas-api-revoke-sub-7cd3b896-bwjhx\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.310061 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.309990    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cd3b896-bwjhx\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.310645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.310617    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cd3b896-bwjhx\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.312320 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.312295    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cd3b896-bwjhx\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.366256 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.366228    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6wcg\" (UniqueName: \"kubernetes.io/projected/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-kube-api-access-p6wcg\") pod \"maas-api-revoke-sub-7cd3b896-bwjhx\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.489347 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.489272    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:54.852926 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:57:54.852892    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa90522a_9b40_40dd_8fe8_f5a48161fe4f.slice/crio-7b2dab97e2b65dcc0c9c91f005aa6161e4b3cacda655b164b309e8867e29baa6 WatchSource:0}: Error finding container 7b2dab97e2b65dcc0c9c91f005aa6161e4b3cacda655b164b309e8867e29baa6: Status 404 returned error can't find the container with id 7b2dab97e2b65dcc0c9c91f005aa6161e4b3cacda655b164b309e8867e29baa6
Nov 02 22:57:54.855717 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:54.855693    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"]
Nov 02 22:57:55.425351 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:55.425316    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx" event={"ID":"aa90522a-9b40-40dd-8fe8-f5a48161fe4f","Type":"ContainerStarted","Data":"a150ebc1633edbcdc56484d0010ea722075b9930c3ef71aabdf4b0e567ee8094"}
Nov 02 22:57:55.425782 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:55.425363    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx" event={"ID":"aa90522a-9b40-40dd-8fe8-f5a48161fe4f","Type":"ContainerStarted","Data":"7b2dab97e2b65dcc0c9c91f005aa6161e4b3cacda655b164b309e8867e29baa6"}
Nov 02 22:57:55.540337 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:55.540288    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx" podStartSLOduration=1.540271628 podStartE2EDuration="1.540271628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:55.539250947 +0000 UTC m=+6562.802787264" watchObservedRunningTime="2026-11-02 22:57:55.540271628 +0000 UTC m=+6562.803807946"
Nov 02 22:57:56.431168 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:56.431130    4528 generic.go:356] "Generic (PLEG): container finished" podID="aa90522a-9b40-40dd-8fe8-f5a48161fe4f" containerID="a150ebc1633edbcdc56484d0010ea722075b9930c3ef71aabdf4b0e567ee8094" exitCode=0
Nov 02 22:57:56.431626 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:56.431193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx" event={"ID":"aa90522a-9b40-40dd-8fe8-f5a48161fe4f","Type":"ContainerDied","Data":"a150ebc1633edbcdc56484d0010ea722075b9930c3ef71aabdf4b0e567ee8094"}
Nov 02 22:57:57.562872 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.562845    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:57.637370 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.637343    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-xks-ca\") pod \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") "
Nov 02 22:57:57.637509 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.637438    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-kube-api-access-p6wcg\" (UniqueName: \"kubernetes.io/projected/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-kube-api-access-p6wcg\") pod \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") "
Nov 02 22:57:57.637553 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.637513    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-openshift-ca\") pod \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\" (UID: \"aa90522a-9b40-40dd-8fe8-f5a48161fe4f\") "
Nov 02 22:57:57.637873 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.637851    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-openshift-ca" pod "aa90522a-9b40-40dd-8fe8-f5a48161fe4f" (UID: "aa90522a-9b40-40dd-8fe8-f5a48161fe4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:57:57.639433 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.639411    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-xks-ca" pod "aa90522a-9b40-40dd-8fe8-f5a48161fe4f" (UID: "aa90522a-9b40-40dd-8fe8-f5a48161fe4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:57:57.639537 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.639447    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-kube-api-access-p6wcg" pod "aa90522a-9b40-40dd-8fe8-f5a48161fe4f" (UID: "aa90522a-9b40-40dd-8fe8-f5a48161fe4f"). InnerVolumeSpecName "kube-api-access-p6wcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:57:57.738541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.738451    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:57:57.738541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.738484    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:57:57.738541 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:57.738500    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6wcg\" (UniqueName: \"kubernetes.io/projected/aa90522a-9b40-40dd-8fe8-f5a48161fe4f-kube-api-access-p6wcg\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:57:58.441586 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:58.441560    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"
Nov 02 22:57:58.441586 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:58.441573    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx" event={"ID":"aa90522a-9b40-40dd-8fe8-f5a48161fe4f","Type":"ContainerDied","Data":"7b2dab97e2b65dcc0c9c91f005aa6161e4b3cacda655b164b309e8867e29baa6"}
Nov 02 22:57:58.441881 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:57:58.441614    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b2dab97e2b65dcc0c9c91f005aa6161e4b3cacda655b164b309e8867e29baa6"
Nov 02 22:58:04.914004 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:04.913973    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"]
Nov 02 22:58:04.914410 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:04.914349    4528 cpu_manager.go:414] "containerMap: removing container" podUID="aa90522a-9b40-40dd-8fe8-f5a48161fe4f" containerName="revoke-keys"
Nov 02 22:58:04.914410 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:04.914361    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa90522a-9b40-40dd-8fe8-f5a48161fe4f" containerName="revoke-keys"
Nov 02 22:58:04.914410 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:04.914373    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy"
Nov 02 22:58:04.914410 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:04.914379    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8d496f-0554-4834-9f38-70d09665112e" containerName="istio-proxy"
Nov 02 22:58:04.914552 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:04.914447    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa90522a-9b40-40dd-8fe8-f5a48161fe4f" containerName="revoke-keys"
Nov 02 22:58:04.917518 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:04.917492    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:04.968381 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:04.968354    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"]
Nov 02 22:58:05.006957 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.006926    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10877a1e-rgkvk\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.007140 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.007010    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-775q6\" (UniqueName: \"kubernetes.io/projected/b25f77e8-8271-447b-b224-77aabd7b1d1d-kube-api-access-775q6\") pod \"maas-api-revoke-sub-10877a1e-rgkvk\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.007215 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.007192    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10877a1e-rgkvk\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.016554 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.016530    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:58:05.016651 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.016584    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:58:05.019728 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.019713    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:58:05.019788 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.019724    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:58:05.108596 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.108570    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-775q6\" (UniqueName: \"kubernetes.io/projected/b25f77e8-8271-447b-b224-77aabd7b1d1d-kube-api-access-775q6\") pod \"maas-api-revoke-sub-10877a1e-rgkvk\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.108741 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.108621    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10877a1e-rgkvk\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.108741 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.108656    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10877a1e-rgkvk\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.109288 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.109267    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10877a1e-rgkvk\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.111131 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.111099    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10877a1e-rgkvk\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.165333 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.165273    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-775q6\" (UniqueName: \"kubernetes.io/projected/b25f77e8-8271-447b-b224-77aabd7b1d1d-kube-api-access-775q6\") pod \"maas-api-revoke-sub-10877a1e-rgkvk\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.229268 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.229244    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:05.592590 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:58:05.592502    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25f77e8_8271_447b_b224_77aabd7b1d1d.slice/crio-b8f8edc2fa7a72c9b15d6f1d85d75ac1a5160f7ee0b4ccecc1c515374aad11e2 WatchSource:0}: Error finding container b8f8edc2fa7a72c9b15d6f1d85d75ac1a5160f7ee0b4ccecc1c515374aad11e2: Status 404 returned error can't find the container with id b8f8edc2fa7a72c9b15d6f1d85d75ac1a5160f7ee0b4ccecc1c515374aad11e2
Nov 02 22:58:05.594064 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:05.594041    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"]
Nov 02 22:58:06.476937 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:06.476902    4528 generic.go:356] "Generic (PLEG): container finished" podID="b25f77e8-8271-447b-b224-77aabd7b1d1d" containerID="8f198949d46f099539d9ab92d1949c9ece2fb994d340e271548e0ff991b907c7" exitCode=0
Nov 02 22:58:06.477324 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:06.476962    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk" event={"ID":"b25f77e8-8271-447b-b224-77aabd7b1d1d","Type":"ContainerDied","Data":"8f198949d46f099539d9ab92d1949c9ece2fb994d340e271548e0ff991b907c7"}
Nov 02 22:58:06.477324 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:06.477010    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk" event={"ID":"b25f77e8-8271-447b-b224-77aabd7b1d1d","Type":"ContainerStarted","Data":"b8f8edc2fa7a72c9b15d6f1d85d75ac1a5160f7ee0b4ccecc1c515374aad11e2"}
Nov 02 22:58:07.612822 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.612801    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:07.732220 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.732144    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-xks-ca\") pod \"b25f77e8-8271-447b-b224-77aabd7b1d1d\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") "
Nov 02 22:58:07.732378 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.732251    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25f77e8-8271-447b-b224-77aabd7b1d1d-kube-api-access-775q6\" (UniqueName: \"kubernetes.io/projected/b25f77e8-8271-447b-b224-77aabd7b1d1d-kube-api-access-775q6\") pod \"b25f77e8-8271-447b-b224-77aabd7b1d1d\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") "
Nov 02 22:58:07.732378 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.732299    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-openshift-ca\") pod \"b25f77e8-8271-447b-b224-77aabd7b1d1d\" (UID: \"b25f77e8-8271-447b-b224-77aabd7b1d1d\") "
Nov 02 22:58:07.732693 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.732668    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-openshift-ca" pod "b25f77e8-8271-447b-b224-77aabd7b1d1d" (UID: "b25f77e8-8271-447b-b224-77aabd7b1d1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:58:07.734356 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.734327    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-xks-ca" pod "b25f77e8-8271-447b-b224-77aabd7b1d1d" (UID: "b25f77e8-8271-447b-b224-77aabd7b1d1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:58:07.734463 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.734406    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25f77e8-8271-447b-b224-77aabd7b1d1d-kube-api-access-775q6" pod "b25f77e8-8271-447b-b224-77aabd7b1d1d" (UID: "b25f77e8-8271-447b-b224-77aabd7b1d1d"). InnerVolumeSpecName "kube-api-access-775q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:58:07.833604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.833578    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:58:07.833604 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.833605    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-775q6\" (UniqueName: \"kubernetes.io/projected/b25f77e8-8271-447b-b224-77aabd7b1d1d-kube-api-access-775q6\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:58:07.833761 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:07.833616    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25f77e8-8271-447b-b224-77aabd7b1d1d-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:58:08.487134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:08.487091    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"
Nov 02 22:58:08.487134 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:08.487125    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk" event={"ID":"b25f77e8-8271-447b-b224-77aabd7b1d1d","Type":"ContainerDied","Data":"b8f8edc2fa7a72c9b15d6f1d85d75ac1a5160f7ee0b4ccecc1c515374aad11e2"}
Nov 02 22:58:08.487349 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:08.487174    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f8edc2fa7a72c9b15d6f1d85d75ac1a5160f7ee0b4ccecc1c515374aad11e2"
Nov 02 22:58:14.111613 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:58:14.111577    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:58:17.550886 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:17.550854    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:58:25.882341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:25.882298    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:58:25.882812 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:25.882369    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:58:25.884253 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:25.884234    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:58:25.884317 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:25.884266    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:58:29.181073 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:29.181034    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:58:38.008129 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.008073    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"]
Nov 02 22:58:38.008542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.008466    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b25f77e8-8271-447b-b224-77aabd7b1d1d" containerName="revoke-keys"
Nov 02 22:58:38.008542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.008479    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25f77e8-8271-447b-b224-77aabd7b1d1d" containerName="revoke-keys"
Nov 02 22:58:38.008542 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.008535    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25f77e8-8271-447b-b224-77aabd7b1d1d" containerName="revoke-keys"
Nov 02 22:58:38.011664 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.011648    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.068552 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.068521    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"]
Nov 02 22:58:38.116506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.116480    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:58:38.116506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.116493    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:58:38.116506 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.116476    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:58:38.116827 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.116549    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:58:38.119926 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.119906    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn2pb\" (UniqueName: \"kubernetes.io/projected/1bce206b-b859-485f-865f-502ea6b24edb-kube-api-access-mn2pb\") pod \"maas-api-revoke-sub-4ceea6ff-gjzj8\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.120039 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.119950    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ceea6ff-gjzj8\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.120079 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.120038    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ceea6ff-gjzj8\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.221455 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.221424    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ceea6ff-gjzj8\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.221618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.221484    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn2pb\" (UniqueName: \"kubernetes.io/projected/1bce206b-b859-485f-865f-502ea6b24edb-kube-api-access-mn2pb\") pod \"maas-api-revoke-sub-4ceea6ff-gjzj8\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.221618 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.221529    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ceea6ff-gjzj8\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.222230 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.222206    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ceea6ff-gjzj8\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.223776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.223752    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ceea6ff-gjzj8\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.278311 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.278282    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn2pb\" (UniqueName: \"kubernetes.io/projected/1bce206b-b859-485f-865f-502ea6b24edb-kube-api-access-mn2pb\") pod \"maas-api-revoke-sub-4ceea6ff-gjzj8\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.324301 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.324273    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:38.688266 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:58:38.688234    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bce206b_b859_485f_865f_502ea6b24edb.slice/crio-735fc4a64b95a4128566eea891779c7973cedd6b4833504932fdabfa65f016ec WatchSource:0}: Error finding container 735fc4a64b95a4128566eea891779c7973cedd6b4833504932fdabfa65f016ec: Status 404 returned error can't find the container with id 735fc4a64b95a4128566eea891779c7973cedd6b4833504932fdabfa65f016ec
Nov 02 22:58:38.690003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:38.689975    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"]
Nov 02 22:58:39.630976 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:39.630943    4528 generic.go:356] "Generic (PLEG): container finished" podID="1bce206b-b859-485f-865f-502ea6b24edb" containerID="b1add172b71c773724ad311416606274b8f4e2cdf764ece2e50c07ae8684fb38" exitCode=0
Nov 02 22:58:39.631431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:39.631035    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8" event={"ID":"1bce206b-b859-485f-865f-502ea6b24edb","Type":"ContainerDied","Data":"b1add172b71c773724ad311416606274b8f4e2cdf764ece2e50c07ae8684fb38"}
Nov 02 22:58:39.631431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:39.631091    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8" event={"ID":"1bce206b-b859-485f-865f-502ea6b24edb","Type":"ContainerStarted","Data":"735fc4a64b95a4128566eea891779c7973cedd6b4833504932fdabfa65f016ec"}
Nov 02 22:58:40.765225 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.765200    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:40.843950 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.843924    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-openshift-ca\") pod \"1bce206b-b859-485f-865f-502ea6b24edb\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") "
Nov 02 22:58:40.844138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.843961    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-xks-ca\") pod \"1bce206b-b859-485f-865f-502ea6b24edb\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") "
Nov 02 22:58:40.844138 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.844032    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bce206b-b859-485f-865f-502ea6b24edb-kube-api-access-mn2pb\" (UniqueName: \"kubernetes.io/projected/1bce206b-b859-485f-865f-502ea6b24edb-kube-api-access-mn2pb\") pod \"1bce206b-b859-485f-865f-502ea6b24edb\" (UID: \"1bce206b-b859-485f-865f-502ea6b24edb\") "
Nov 02 22:58:40.844369 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.844346    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-openshift-ca" pod "1bce206b-b859-485f-865f-502ea6b24edb" (UID: "1bce206b-b859-485f-865f-502ea6b24edb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:58:40.844462 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.844449    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:58:40.845990 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.845954    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-xks-ca" pod "1bce206b-b859-485f-865f-502ea6b24edb" (UID: "1bce206b-b859-485f-865f-502ea6b24edb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:58:40.846133 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.846058    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bce206b-b859-485f-865f-502ea6b24edb-kube-api-access-mn2pb" pod "1bce206b-b859-485f-865f-502ea6b24edb" (UID: "1bce206b-b859-485f-865f-502ea6b24edb"). InnerVolumeSpecName "kube-api-access-mn2pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:58:40.945202 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.945167    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bce206b-b859-485f-865f-502ea6b24edb-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:58:40.945202 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:40.945198    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn2pb\" (UniqueName: \"kubernetes.io/projected/1bce206b-b859-485f-865f-502ea6b24edb-kube-api-access-mn2pb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:58:41.644890 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:41.644862    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"
Nov 02 22:58:41.645076 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:41.644857    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8" event={"ID":"1bce206b-b859-485f-865f-502ea6b24edb","Type":"ContainerDied","Data":"735fc4a64b95a4128566eea891779c7973cedd6b4833504932fdabfa65f016ec"}
Nov 02 22:58:41.645076 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:41.645011    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="735fc4a64b95a4128566eea891779c7973cedd6b4833504932fdabfa65f016ec"
Nov 02 22:58:44.112365 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:58:44.112329    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:58:50.663378 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:58:50.663346    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:59:02.362816 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:02.362781    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:59:11.046702 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:11.046663    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"]
Nov 02 22:59:11.096823 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:11.096787    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c04135-j6wvm"]
Nov 02 22:59:12.911948 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:12.911909    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a7c4da-1fa7-4f75-8298-f857cdf5cddb" path="/var/lib/kubelet/pods/31a7c4da-1fa7-4f75-8298-f857cdf5cddb/volumes"
Nov 02 22:59:14.112284 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:59:14.112249    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:59:20.300052 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.300018    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"]
Nov 02 22:59:20.300449 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.300402    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1bce206b-b859-485f-865f-502ea6b24edb" containerName="revoke-keys"
Nov 02 22:59:20.300449 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.300413    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bce206b-b859-485f-865f-502ea6b24edb" containerName="revoke-keys"
Nov 02 22:59:20.300527 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.300466    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bce206b-b859-485f-865f-502ea6b24edb" containerName="revoke-keys"
Nov 02 22:59:20.304669 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.304652    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.346023 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.346000    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd6zk\" (UniqueName: \"kubernetes.io/projected/708ccd0d-118a-40fb-ba51-9ddc657cb131-kube-api-access-xd6zk\") pod \"maas-api-revoke-sub-43286820-8hgzf\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.346177 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.346034    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43286820-8hgzf\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.346177 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.346152    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43286820-8hgzf\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.356410 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.356385    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"]
Nov 02 22:59:20.404819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.404792    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:59:20.404819 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.404792    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:59:20.406956 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.406939    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:59:20.407016 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.406985    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:59:20.446685 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.446659    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd6zk\" (UniqueName: \"kubernetes.io/projected/708ccd0d-118a-40fb-ba51-9ddc657cb131-kube-api-access-xd6zk\") pod \"maas-api-revoke-sub-43286820-8hgzf\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.446841 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.446696    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43286820-8hgzf\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.446841 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.446738    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43286820-8hgzf\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.447348 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.447330    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43286820-8hgzf\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.449013 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.448995    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43286820-8hgzf\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.503324 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.503302    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd6zk\" (UniqueName: \"kubernetes.io/projected/708ccd0d-118a-40fb-ba51-9ddc657cb131-kube-api-access-xd6zk\") pod \"maas-api-revoke-sub-43286820-8hgzf\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.617003 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.616928    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:20.983190 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:59:20.983158    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708ccd0d_118a_40fb_ba51_9ddc657cb131.slice/crio-067cc8af96919f9b356d11215b4dcc16f6161878e1adce810a8fe7eb622df17d WatchSource:0}: Error finding container 067cc8af96919f9b356d11215b4dcc16f6161878e1adce810a8fe7eb622df17d: Status 404 returned error can't find the container with id 067cc8af96919f9b356d11215b4dcc16f6161878e1adce810a8fe7eb622df17d
Nov 02 22:59:20.984548 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:20.984526    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"]
Nov 02 22:59:21.848228 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:21.848193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf" event={"ID":"708ccd0d-118a-40fb-ba51-9ddc657cb131","Type":"ContainerStarted","Data":"33e3e6e67f21e76d9a889fe896d2da3c6e71d99d3377f3006273f1d5f6d57958"}
Nov 02 22:59:21.848697 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:21.848251    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf" event={"ID":"708ccd0d-118a-40fb-ba51-9ddc657cb131","Type":"ContainerStarted","Data":"067cc8af96919f9b356d11215b4dcc16f6161878e1adce810a8fe7eb622df17d"}
Nov 02 22:59:22.013664 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:22.013613    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf" podStartSLOduration=2.013600547 podStartE2EDuration="2.013600547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:21.963047463 +0000 UTC m=+6649.226583779" watchObservedRunningTime="2026-11-02 22:59:22.013600547 +0000 UTC m=+6649.277136863"
Nov 02 22:59:22.854370 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:22.854333    4528 generic.go:356] "Generic (PLEG): container finished" podID="708ccd0d-118a-40fb-ba51-9ddc657cb131" containerID="33e3e6e67f21e76d9a889fe896d2da3c6e71d99d3377f3006273f1d5f6d57958" exitCode=0
Nov 02 22:59:22.854776 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:22.854411    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf" event={"ID":"708ccd0d-118a-40fb-ba51-9ddc657cb131","Type":"ContainerDied","Data":"33e3e6e67f21e76d9a889fe896d2da3c6e71d99d3377f3006273f1d5f6d57958"}
Nov 02 22:59:23.987484 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:23.987459    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:24.081984 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.081957    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-openshift-ca\") pod \"708ccd0d-118a-40fb-ba51-9ddc657cb131\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") "
Nov 02 22:59:24.082163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.082025    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708ccd0d-118a-40fb-ba51-9ddc657cb131-kube-api-access-xd6zk\" (UniqueName: \"kubernetes.io/projected/708ccd0d-118a-40fb-ba51-9ddc657cb131-kube-api-access-xd6zk\") pod \"708ccd0d-118a-40fb-ba51-9ddc657cb131\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") "
Nov 02 22:59:24.082163 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.082146    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-xks-ca\") pod \"708ccd0d-118a-40fb-ba51-9ddc657cb131\" (UID: \"708ccd0d-118a-40fb-ba51-9ddc657cb131\") "
Nov 02 22:59:24.082385 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.082357    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-openshift-ca" pod "708ccd0d-118a-40fb-ba51-9ddc657cb131" (UID: "708ccd0d-118a-40fb-ba51-9ddc657cb131"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:59:24.084031 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.084007    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-xks-ca" pod "708ccd0d-118a-40fb-ba51-9ddc657cb131" (UID: "708ccd0d-118a-40fb-ba51-9ddc657cb131"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:59:24.084146 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.084047    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708ccd0d-118a-40fb-ba51-9ddc657cb131-kube-api-access-xd6zk" pod "708ccd0d-118a-40fb-ba51-9ddc657cb131" (UID: "708ccd0d-118a-40fb-ba51-9ddc657cb131"). InnerVolumeSpecName "kube-api-access-xd6zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:59:24.183341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.183271    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:24.183341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.183294    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd6zk\" (UniqueName: \"kubernetes.io/projected/708ccd0d-118a-40fb-ba51-9ddc657cb131-kube-api-access-xd6zk\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:24.183341 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.183305    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708ccd0d-118a-40fb-ba51-9ddc657cb131-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:24.866100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.866065    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf" event={"ID":"708ccd0d-118a-40fb-ba51-9ddc657cb131","Type":"ContainerDied","Data":"067cc8af96919f9b356d11215b4dcc16f6161878e1adce810a8fe7eb622df17d"}
Nov 02 22:59:24.866100 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.866098    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"
Nov 02 22:59:24.866338 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:24.866133    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="067cc8af96919f9b356d11215b4dcc16f6161878e1adce810a8fe7eb622df17d"
Nov 02 22:59:31.135205 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.135170    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"]
Nov 02 22:59:31.135732 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.135712    4528 cpu_manager.go:414] "containerMap: removing container" podUID="708ccd0d-118a-40fb-ba51-9ddc657cb131" containerName="revoke-keys"
Nov 02 22:59:31.135813 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.135735    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708ccd0d-118a-40fb-ba51-9ddc657cb131" containerName="revoke-keys"
Nov 02 22:59:31.135880 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.135839    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="708ccd0d-118a-40fb-ba51-9ddc657cb131" containerName="revoke-keys"
Nov 02 22:59:31.138806 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.138785    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.191790 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.191756    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"]
Nov 02 22:59:31.239584 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.239553    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 22:59:31.239777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.239606    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:59:31.239777 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.239638    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 22:59:31.241178 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.241161    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:59:31.245603 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.245580    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26f2aba-46ck9\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.245720 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.245617    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26f2aba-46ck9\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.245720 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.245646    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrvb\" (UniqueName: \"kubernetes.io/projected/57d07466-7e06-4094-987e-a9ac0d9d3723-kube-api-access-gbrvb\") pod \"maas-api-revoke-sub-e26f2aba-46ck9\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.346494 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.346463    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26f2aba-46ck9\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.346668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.346503    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26f2aba-46ck9\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.346668 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.346524    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrvb\" (UniqueName: \"kubernetes.io/projected/57d07466-7e06-4094-987e-a9ac0d9d3723-kube-api-access-gbrvb\") pod \"maas-api-revoke-sub-e26f2aba-46ck9\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.347187 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.347160    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26f2aba-46ck9\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.348833 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.348815    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26f2aba-46ck9\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.402988 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.402928    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrvb\" (UniqueName: \"kubernetes.io/projected/57d07466-7e06-4094-987e-a9ac0d9d3723-kube-api-access-gbrvb\") pod \"maas-api-revoke-sub-e26f2aba-46ck9\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.452402 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.452372    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:31.814859 ip-10-0-133-86 kubenswrapper[4528]: W1102 22:59:31.814816    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d07466_7e06_4094_987e_a9ac0d9d3723.slice/crio-39d407753adf2ffcbd4b20bf01dcf1ab8b50e739c3d8e65a88caf0364c2e42cb WatchSource:0}: Error finding container 39d407753adf2ffcbd4b20bf01dcf1ab8b50e739c3d8e65a88caf0364c2e42cb: Status 404 returned error can't find the container with id 39d407753adf2ffcbd4b20bf01dcf1ab8b50e739c3d8e65a88caf0364c2e42cb
Nov 02 22:59:31.816572 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.816543    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"]
Nov 02 22:59:31.897866 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:31.897837    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9" event={"ID":"57d07466-7e06-4094-987e-a9ac0d9d3723","Type":"ContainerStarted","Data":"39d407753adf2ffcbd4b20bf01dcf1ab8b50e739c3d8e65a88caf0364c2e42cb"}
Nov 02 22:59:32.903995 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:32.903913    4528 generic.go:356] "Generic (PLEG): container finished" podID="57d07466-7e06-4094-987e-a9ac0d9d3723" containerID="a7f4ac798541bfa569146776855d664154e6fe46fb6ccc5001590f9f54bcb913" exitCode=0
Nov 02 22:59:32.904392 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:32.903989    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9" event={"ID":"57d07466-7e06-4094-987e-a9ac0d9d3723","Type":"ContainerDied","Data":"a7f4ac798541bfa569146776855d664154e6fe46fb6ccc5001590f9f54bcb913"}
Nov 02 22:59:33.047470 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:33.040594    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:59:34.048368 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.048344    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:34.172444 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.172364    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-openshift-ca\") pod \"57d07466-7e06-4094-987e-a9ac0d9d3723\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") "
Nov 02 22:59:34.172622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.172445    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d07466-7e06-4094-987e-a9ac0d9d3723-kube-api-access-gbrvb\" (UniqueName: \"kubernetes.io/projected/57d07466-7e06-4094-987e-a9ac0d9d3723-kube-api-access-gbrvb\") pod \"57d07466-7e06-4094-987e-a9ac0d9d3723\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") "
Nov 02 22:59:34.172622 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.172480    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-xks-ca\") pod \"57d07466-7e06-4094-987e-a9ac0d9d3723\" (UID: \"57d07466-7e06-4094-987e-a9ac0d9d3723\") "
Nov 02 22:59:34.172762 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.172715    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-openshift-ca" pod "57d07466-7e06-4094-987e-a9ac0d9d3723" (UID: "57d07466-7e06-4094-987e-a9ac0d9d3723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:59:34.174499 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.174473    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d07466-7e06-4094-987e-a9ac0d9d3723-kube-api-access-gbrvb" pod "57d07466-7e06-4094-987e-a9ac0d9d3723" (UID: "57d07466-7e06-4094-987e-a9ac0d9d3723"). InnerVolumeSpecName "kube-api-access-gbrvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:59:34.174645 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.174499    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-xks-ca" pod "57d07466-7e06-4094-987e-a9ac0d9d3723" (UID: "57d07466-7e06-4094-987e-a9ac0d9d3723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:59:34.273431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.273399    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:34.273431 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.273426    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbrvb\" (UniqueName: \"kubernetes.io/projected/57d07466-7e06-4094-987e-a9ac0d9d3723-kube-api-access-gbrvb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:34.273658 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.273441    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d07466-7e06-4094-987e-a9ac0d9d3723-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:34.917024 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.916994    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"
Nov 02 22:59:34.917223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.917026    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9" event={"ID":"57d07466-7e06-4094-987e-a9ac0d9d3723","Type":"ContainerDied","Data":"39d407753adf2ffcbd4b20bf01dcf1ab8b50e739c3d8e65a88caf0364c2e42cb"}
Nov 02 22:59:34.917223 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:34.917064    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39d407753adf2ffcbd4b20bf01dcf1ab8b50e739c3d8e65a88caf0364c2e42cb"
Nov 02 22:59:43.857424 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:43.857387    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 22:59:44.113083 ip-10-0-133-86 kubenswrapper[4528]: E1102 22:59:44.112984    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 22:59:57.374220 ip-10-0-133-86 kubenswrapper[4528]: I1102 22:59:57.374172    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 23:00:00.165525 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.165491    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"]
Nov 02 23:00:00.165937 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.165905    4528 cpu_manager.go:414] "containerMap: removing container" podUID="57d07466-7e06-4094-987e-a9ac0d9d3723" containerName="revoke-keys"
Nov 02 23:00:00.165937 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.165919    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d07466-7e06-4094-987e-a9ac0d9d3723" containerName="revoke-keys"
Nov 02 23:00:00.166023 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.165987    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d07466-7e06-4094-987e-a9ac0d9d3723" containerName="revoke-keys"
Nov 02 23:00:00.169098 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.169076    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:00.221298 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.221266    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"]
Nov 02 23:00:00.269054 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.269022    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:00:00.269054 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.269022    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:00:00.269054 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.269052    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:00:00.278727 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.278703    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f962de-1563-4b6a-bf63-478fcd3012c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-rdppx\" (UID: \"27f962de-1563-4b6a-bf63-478fcd3012c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:00.278727 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.278731    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wcb\" (UniqueName: \"kubernetes.io/projected/27f962de-1563-4b6a-bf63-478fcd3012c5-kube-api-access-94wcb\") pod \"maas-api-key-cleanup-29894340-rdppx\" (UID: \"27f962de-1563-4b6a-bf63-478fcd3012c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:00.379534 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.379504    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f962de-1563-4b6a-bf63-478fcd3012c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-rdppx\" (UID: \"27f962de-1563-4b6a-bf63-478fcd3012c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:00.379534 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.379537    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wcb\" (UniqueName: \"kubernetes.io/projected/27f962de-1563-4b6a-bf63-478fcd3012c5-kube-api-access-94wcb\") pod \"maas-api-key-cleanup-29894340-rdppx\" (UID: \"27f962de-1563-4b6a-bf63-478fcd3012c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:00.380191 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.380166    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f962de-1563-4b6a-bf63-478fcd3012c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-rdppx\" (UID: \"27f962de-1563-4b6a-bf63-478fcd3012c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:00.436131 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.436056    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wcb\" (UniqueName: \"kubernetes.io/projected/27f962de-1563-4b6a-bf63-478fcd3012c5-kube-api-access-94wcb\") pod \"maas-api-key-cleanup-29894340-rdppx\" (UID: \"27f962de-1563-4b6a-bf63-478fcd3012c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:00.486282 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.486255    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:00.847750 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:00:00.847716    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f962de_1563_4b6a_bf63_478fcd3012c5.slice/crio-f562e313c4a743c2af07a43f081df68e4a996ccae4ef5eacfa136a2aac64dce8 WatchSource:0}: Error finding container f562e313c4a743c2af07a43f081df68e4a996ccae4ef5eacfa136a2aac64dce8: Status 404 returned error can't find the container with id f562e313c4a743c2af07a43f081df68e4a996ccae4ef5eacfa136a2aac64dce8
Nov 02 23:00:00.849476 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:00.849450    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"]
Nov 02 23:00:01.051087 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:01.051060    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx" event={"ID":"27f962de-1563-4b6a-bf63-478fcd3012c5","Type":"ContainerStarted","Data":"f562e313c4a743c2af07a43f081df68e4a996ccae4ef5eacfa136a2aac64dce8"}
Nov 02 23:00:02.057787 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:02.057753    4528 generic.go:356] "Generic (PLEG): container finished" podID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerID="86ad081099a7027fa560d1001c8e96334f7f8ba7467111763fea66ace9a50570" exitCode=60
Nov 02 23:00:02.058224 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:02.057835    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx" event={"ID":"27f962de-1563-4b6a-bf63-478fcd3012c5","Type":"ContainerDied","Data":"86ad081099a7027fa560d1001c8e96334f7f8ba7467111763fea66ace9a50570"}
Nov 02 23:00:02.058224 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:02.058030    4528 scope.go:122] "RemoveContainer" containerID="86ad081099a7027fa560d1001c8e96334f7f8ba7467111763fea66ace9a50570"
Nov 02 23:00:03.064443 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:03.064350    4528 generic.go:356] "Generic (PLEG): container finished" podID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerID="9b69934730190cf03f1c7df7a00eab769c9ce524c759dd64756065e1f9cc2c37" exitCode=60
Nov 02 23:00:03.064942 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:03.064437    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx" event={"ID":"27f962de-1563-4b6a-bf63-478fcd3012c5","Type":"ContainerDied","Data":"9b69934730190cf03f1c7df7a00eab769c9ce524c759dd64756065e1f9cc2c37"}
Nov 02 23:00:03.064942 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:03.064501    4528 scope.go:122] "RemoveContainer" containerID="86ad081099a7027fa560d1001c8e96334f7f8ba7467111763fea66ace9a50570"
Nov 02 23:00:03.064942 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:03.064630    4528 scope.go:122] "RemoveContainer" containerID="9b69934730190cf03f1c7df7a00eab769c9ce524c759dd64756065e1f9cc2c37"
Nov 02 23:00:03.064942 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:00:03.064878    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-rdppx_odh-ai-gateway-infra(27f962de-1563-4b6a-bf63-478fcd3012c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5"
Nov 02 23:00:04.071567 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:04.071540    4528 scope.go:122] "RemoveContainer" containerID="9b69934730190cf03f1c7df7a00eab769c9ce524c759dd64756065e1f9cc2c37"
Nov 02 23:00:04.071955 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:00:04.071772    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-rdppx_odh-ai-gateway-infra(27f962de-1563-4b6a-bf63-478fcd3012c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5"
Nov 02 23:00:08.180839 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:08.180807    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 23:00:10.877335 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:10.877286    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:00:10.877873 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:10.877360    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:00:10.877873 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:10.877813    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:00:10.878019 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:10.877868    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:00:13.906603 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:13.906570    4528 scope.go:122] "RemoveContainer" containerID="9b69934730190cf03f1c7df7a00eab769c9ce524c759dd64756065e1f9cc2c37"
Nov 02 23:00:14.112579 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:00:14.112544    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:00:15.128980 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:15.128944    4528 generic.go:356] "Generic (PLEG): container finished" podID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerID="6d8b298a6b87bf6dc9c25301d0e3851853177fcc5afd001130e5dfb22585cb2c" exitCode=60
Nov 02 23:00:15.129505 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:15.129014    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx" event={"ID":"27f962de-1563-4b6a-bf63-478fcd3012c5","Type":"ContainerDied","Data":"6d8b298a6b87bf6dc9c25301d0e3851853177fcc5afd001130e5dfb22585cb2c"}
Nov 02 23:00:15.129505 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:15.129069    4528 scope.go:122] "RemoveContainer" containerID="9b69934730190cf03f1c7df7a00eab769c9ce524c759dd64756065e1f9cc2c37"
Nov 02 23:00:15.129505 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:15.129491    4528 scope.go:122] "RemoveContainer" containerID="6d8b298a6b87bf6dc9c25301d0e3851853177fcc5afd001130e5dfb22585cb2c"
Nov 02 23:00:15.129792 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:00:15.129770    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-rdppx_odh-ai-gateway-infra(27f962de-1563-4b6a-bf63-478fcd3012c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5"
Nov 02 23:00:16.253166 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:16.253134    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"]
Nov 02 23:00:16.396449 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:16.396427    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:16.443767 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:16.443739    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f962de-1563-4b6a-bf63-478fcd3012c5-kube-api-access-94wcb\" (UniqueName: \"kubernetes.io/projected/27f962de-1563-4b6a-bf63-478fcd3012c5-kube-api-access-94wcb\") pod \"27f962de-1563-4b6a-bf63-478fcd3012c5\" (UID: \"27f962de-1563-4b6a-bf63-478fcd3012c5\") "
Nov 02 23:00:16.443924 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:16.443800    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f962de-1563-4b6a-bf63-478fcd3012c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f962de-1563-4b6a-bf63-478fcd3012c5-maas-api-service-ca\") pod \"27f962de-1563-4b6a-bf63-478fcd3012c5\" (UID: \"27f962de-1563-4b6a-bf63-478fcd3012c5\") "
Nov 02 23:00:16.444088 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:16.444064    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f962de-1563-4b6a-bf63-478fcd3012c5-maas-api-service-ca" pod "27f962de-1563-4b6a-bf63-478fcd3012c5" (UID: "27f962de-1563-4b6a-bf63-478fcd3012c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:00:16.445785 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:16.445766    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f962de-1563-4b6a-bf63-478fcd3012c5-kube-api-access-94wcb" pod "27f962de-1563-4b6a-bf63-478fcd3012c5" (UID: "27f962de-1563-4b6a-bf63-478fcd3012c5"). InnerVolumeSpecName "kube-api-access-94wcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:00:16.545455 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:16.545425    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94wcb\" (UniqueName: \"kubernetes.io/projected/27f962de-1563-4b6a-bf63-478fcd3012c5-kube-api-access-94wcb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:00:16.545455 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:16.545454    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f962de-1563-4b6a-bf63-478fcd3012c5-maas-api-service-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:00:17.143563 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:17.143530    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx" event={"ID":"27f962de-1563-4b6a-bf63-478fcd3012c5","Type":"ContainerDied","Data":"f562e313c4a743c2af07a43f081df68e4a996ccae4ef5eacfa136a2aac64dce8"}
Nov 02 23:00:17.143563 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:17.143553    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"
Nov 02 23:00:17.143808 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:17.143597    4528 scope.go:122] "RemoveContainer" containerID="6d8b298a6b87bf6dc9c25301d0e3851853177fcc5afd001130e5dfb22585cb2c"
Nov 02 23:00:17.307137 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:17.307095    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"]
Nov 02 23:00:17.309013 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:17.308986    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-rdppx"]
Nov 02 23:00:18.827767 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.827735    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"]
Nov 02 23:00:18.828197 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.828118    4528 cpu_manager.go:414] "containerMap: removing container" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerName="cleanup"
Nov 02 23:00:18.828197 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.828132    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerName="cleanup"
Nov 02 23:00:18.828197 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.828149    4528 cpu_manager.go:414] "containerMap: removing container" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerName="cleanup"
Nov 02 23:00:18.828197 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.828154    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerName="cleanup"
Nov 02 23:00:18.828374 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.828223    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerName="cleanup"
Nov 02 23:00:18.828374 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.828234    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerName="cleanup"
Nov 02 23:00:18.832581 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.832566    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:18.866879 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.866851    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2hc\" (UniqueName: \"kubernetes.io/projected/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-kube-api-access-ss2hc\") pod \"maas-api-revoke-sub-3ab714c1-s8hwr\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:18.866984 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.866893    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ab714c1-s8hwr\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:18.867032 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.866987    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ab714c1-s8hwr\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:18.883618 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.883590    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"]
Nov 02 23:00:18.912127 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.912072    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" path="/var/lib/kubelet/pods/27f962de-1563-4b6a-bf63-478fcd3012c5/volumes"
Nov 02 23:00:18.932034 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.931998    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:00:18.932034 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.932016    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:00:18.932034 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.932009    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:00:18.932284 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.932074    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:00:18.967651 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.967625    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ab714c1-s8hwr\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:18.967932 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.967909    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2hc\" (UniqueName: \"kubernetes.io/projected/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-kube-api-access-ss2hc\") pod \"maas-api-revoke-sub-3ab714c1-s8hwr\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:18.968036 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.967954    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ab714c1-s8hwr\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:18.968597 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.968576    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ab714c1-s8hwr\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:18.969988 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:18.969966    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ab714c1-s8hwr\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:19.024812 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:19.024782    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2hc\" (UniqueName: \"kubernetes.io/projected/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-kube-api-access-ss2hc\") pod \"maas-api-revoke-sub-3ab714c1-s8hwr\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:19.144892 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:19.144814    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:19.508925 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:00:19.508888    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3daecb0_ca6c_4b4c_aac4_ef17a9417a1a.slice/crio-c8988a05274c4a15e0a40b17b04a1d5c810304bf93966e81db6c754666602e58 WatchSource:0}: Error finding container c8988a05274c4a15e0a40b17b04a1d5c810304bf93966e81db6c754666602e58: Status 404 returned error can't find the container with id c8988a05274c4a15e0a40b17b04a1d5c810304bf93966e81db6c754666602e58
Nov 02 23:00:19.510930 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:19.510902    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"]
Nov 02 23:00:20.178266 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:20.178185    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr" event={"ID":"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a","Type":"ContainerStarted","Data":"3c29b27762e25a74226ce5a76e14abebca961a8fa2a4a48553beb0b778ecde85"}
Nov 02 23:00:20.178266 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:20.178245    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr" event={"ID":"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a","Type":"ContainerStarted","Data":"c8988a05274c4a15e0a40b17b04a1d5c810304bf93966e81db6c754666602e58"}
Nov 02 23:00:20.292289 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:20.292240    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr" podStartSLOduration=2.29222522 podStartE2EDuration="2.29222522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:20.290775955 +0000 UTC m=+6707.554312271" watchObservedRunningTime="2026-11-02 23:00:20.29222522 +0000 UTC m=+6707.555761536"
Nov 02 23:00:21.185842 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:21.184931    4528 generic.go:356] "Generic (PLEG): container finished" podID="d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a" containerID="3c29b27762e25a74226ce5a76e14abebca961a8fa2a4a48553beb0b778ecde85" exitCode=0
Nov 02 23:00:21.185842 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:21.185090    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr" event={"ID":"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a","Type":"ContainerDied","Data":"3c29b27762e25a74226ce5a76e14abebca961a8fa2a4a48553beb0b778ecde85"}
Nov 02 23:00:22.322221 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.322195    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:22.399923 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.399894    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-kube-api-access-ss2hc\" (UniqueName: \"kubernetes.io/projected/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-kube-api-access-ss2hc\") pod \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") "
Nov 02 23:00:22.400094 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.399990    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-openshift-ca\") pod \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") "
Nov 02 23:00:22.400094 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.400070    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-xks-ca\") pod \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\" (UID: \"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a\") "
Nov 02 23:00:22.400404 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.400377    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-openshift-ca" pod "d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a" (UID: "d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:00:22.401969 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.401945    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-xks-ca" pod "d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a" (UID: "d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:00:22.402069 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.401999    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-kube-api-access-ss2hc" pod "d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a" (UID: "d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a"). InnerVolumeSpecName "kube-api-access-ss2hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:00:22.501276 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.501196    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:00:22.501276 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.501224    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:00:22.501276 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:22.501234    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss2hc\" (UniqueName: \"kubernetes.io/projected/d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a-kube-api-access-ss2hc\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:00:23.196035 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:23.195965    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr"
Nov 02 23:00:23.196207 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:23.195959    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab714c1-s8hwr" event={"ID":"d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a","Type":"ContainerDied","Data":"c8988a05274c4a15e0a40b17b04a1d5c810304bf93966e81db6c754666602e58"}
Nov 02 23:00:23.196207 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:23.196089    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8988a05274c4a15e0a40b17b04a1d5c810304bf93966e81db6c754666602e58"
Nov 02 23:00:29.617748 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.617713    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"]
Nov 02 23:00:29.618361 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.618104    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a" containerName="revoke-keys"
Nov 02 23:00:29.618361 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.618132    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a" containerName="revoke-keys"
Nov 02 23:00:29.618361 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.618149    4528 cpu_manager.go:414] "containerMap: removing container" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerName="cleanup"
Nov 02 23:00:29.618361 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.618154    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerName="cleanup"
Nov 02 23:00:29.618361 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.618219    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f962de-1563-4b6a-bf63-478fcd3012c5" containerName="cleanup"
Nov 02 23:00:29.618361 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.618229    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a" containerName="revoke-keys"
Nov 02 23:00:29.621126 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.621091    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.673359 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.673321    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"]
Nov 02 23:00:29.721563 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.721530    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:00:29.721736 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.721590    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:00:29.721736 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.721624    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:00:29.721736 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.721661    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:00:29.764227 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.764200    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kw2t\" (UniqueName: \"kubernetes.io/projected/4dcba5b8-a32a-41d7-96fb-6161baea6fea-kube-api-access-6kw2t\") pod \"maas-api-revoke-sub-b79f698c-chn77\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.764381 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.764234    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b79f698c-chn77\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.764381 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.764266    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b79f698c-chn77\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.864940 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.864905    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kw2t\" (UniqueName: \"kubernetes.io/projected/4dcba5b8-a32a-41d7-96fb-6161baea6fea-kube-api-access-6kw2t\") pod \"maas-api-revoke-sub-b79f698c-chn77\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.865089 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.864947    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b79f698c-chn77\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.865089 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.864977    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b79f698c-chn77\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.865589 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.865562    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b79f698c-chn77\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.867258 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.867241    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b79f698c-chn77\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.921231 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.921176    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kw2t\" (UniqueName: \"kubernetes.io/projected/4dcba5b8-a32a-41d7-96fb-6161baea6fea-kube-api-access-6kw2t\") pod \"maas-api-revoke-sub-b79f698c-chn77\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:29.938949 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:29.938925    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:30.304020 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:00:30.303986    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcba5b8_a32a_41d7_96fb_6161baea6fea.slice/crio-325643cda4133d841f0945b4e00fb18033a27f826dffb818d7da696229f661d1 WatchSource:0}: Error finding container 325643cda4133d841f0945b4e00fb18033a27f826dffb818d7da696229f661d1: Status 404 returned error can't find the container with id 325643cda4133d841f0945b4e00fb18033a27f826dffb818d7da696229f661d1
Nov 02 23:00:30.306045 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:30.306021    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"]
Nov 02 23:00:31.234370 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:31.234335    4528 generic.go:356] "Generic (PLEG): container finished" podID="4dcba5b8-a32a-41d7-96fb-6161baea6fea" containerID="f95ba2b91fd9c848b90f50699a4e710a4cb788dcb79fb01ad186df52a4411bb0" exitCode=0
Nov 02 23:00:31.234730 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:31.234413    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77" event={"ID":"4dcba5b8-a32a-41d7-96fb-6161baea6fea","Type":"ContainerDied","Data":"f95ba2b91fd9c848b90f50699a4e710a4cb788dcb79fb01ad186df52a4411bb0"}
Nov 02 23:00:31.234730 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:31.234467    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77" event={"ID":"4dcba5b8-a32a-41d7-96fb-6161baea6fea","Type":"ContainerStarted","Data":"325643cda4133d841f0945b4e00fb18033a27f826dffb818d7da696229f661d1"}
Nov 02 23:00:31.576809 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:31.576776    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 23:00:32.401415 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.401388    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:32.490919 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.490838    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dcba5b8-a32a-41d7-96fb-6161baea6fea-kube-api-access-6kw2t\" (UniqueName: \"kubernetes.io/projected/4dcba5b8-a32a-41d7-96fb-6161baea6fea-kube-api-access-6kw2t\") pod \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") "
Nov 02 23:00:32.491062 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.490937    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-openshift-ca\") pod \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") "
Nov 02 23:00:32.491160 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.491097    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-xks-ca\") pod \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\" (UID: \"4dcba5b8-a32a-41d7-96fb-6161baea6fea\") "
Nov 02 23:00:32.491374 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.491341    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-openshift-ca" pod "4dcba5b8-a32a-41d7-96fb-6161baea6fea" (UID: "4dcba5b8-a32a-41d7-96fb-6161baea6fea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:00:32.493204 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.493176    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-xks-ca" pod "4dcba5b8-a32a-41d7-96fb-6161baea6fea" (UID: "4dcba5b8-a32a-41d7-96fb-6161baea6fea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:00:32.493489 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.493471    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcba5b8-a32a-41d7-96fb-6161baea6fea-kube-api-access-6kw2t" pod "4dcba5b8-a32a-41d7-96fb-6161baea6fea" (UID: "4dcba5b8-a32a-41d7-96fb-6161baea6fea"). InnerVolumeSpecName "kube-api-access-6kw2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:00:32.592363 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.592322    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:00:32.592363 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.592361    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kw2t\" (UniqueName: \"kubernetes.io/projected/4dcba5b8-a32a-41d7-96fb-6161baea6fea-kube-api-access-6kw2t\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:00:32.592576 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:32.592379    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dcba5b8-a32a-41d7-96fb-6161baea6fea-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:00:33.262195 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:33.260757    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77" event={"ID":"4dcba5b8-a32a-41d7-96fb-6161baea6fea","Type":"ContainerDied","Data":"325643cda4133d841f0945b4e00fb18033a27f826dffb818d7da696229f661d1"}
Nov 02 23:00:33.262195 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:33.260814    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="325643cda4133d841f0945b4e00fb18033a27f826dffb818d7da696229f661d1"
Nov 02 23:00:33.262195 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:33.260882    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79f698c-chn77"
Nov 02 23:00:37.354799 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:37.354769    4528 scope.go:122] "RemoveContainer" containerID="3ca996a1243403b6b00a92e6ef14a146b11a753cbb11796e5efaa9175b363e07"
Nov 02 23:00:42.254512 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:42.254481    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 23:00:44.173949 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:00:44.173907    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:00:51.477662 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:00:51.477627    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 23:01:07.339134 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:07.338197    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pwm9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:01:07.339134 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:07.338271    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" podUID="00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:01:11.345129 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:11.345076    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 23:01:14.176564 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:01:14.176521    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:01:16.954427 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:16.954391    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"]
Nov 02 23:01:16.954825 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:16.954784    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4dcba5b8-a32a-41d7-96fb-6161baea6fea" containerName="revoke-keys"
Nov 02 23:01:16.954825 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:16.954796    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dcba5b8-a32a-41d7-96fb-6161baea6fea" containerName="revoke-keys"
Nov 02 23:01:16.954908 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:16.954864    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dcba5b8-a32a-41d7-96fb-6161baea6fea" containerName="revoke-keys"
Nov 02 23:01:16.958172 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:16.958154    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.012783 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.012756    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"]
Nov 02 23:01:17.060778 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.060749    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:01:17.060778 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.060772    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:01:17.060984 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.060787    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:01:17.061626 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.061608    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:01:17.102178 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.102152    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtp2\" (UniqueName: \"kubernetes.io/projected/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-kube-api-access-swtp2\") pod \"maas-api-revoke-sub-a162e8e3-nrmbk\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.102292 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.102203    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a162e8e3-nrmbk\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.102369 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.102344    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a162e8e3-nrmbk\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.202936 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.202906    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtp2\" (UniqueName: \"kubernetes.io/projected/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-kube-api-access-swtp2\") pod \"maas-api-revoke-sub-a162e8e3-nrmbk\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.203101 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.202968    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a162e8e3-nrmbk\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.203101 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.203025    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a162e8e3-nrmbk\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.203759 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.203733    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a162e8e3-nrmbk\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.205487 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.205428    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a162e8e3-nrmbk\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.260166 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.260142    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtp2\" (UniqueName: \"kubernetes.io/projected/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-kube-api-access-swtp2\") pod \"maas-api-revoke-sub-a162e8e3-nrmbk\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.271324 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.271303    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:17.636550 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:01:17.636516    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8d95d4_fa9d_4d3c_ab27_03daefc5f78e.slice/crio-275e6fcc8013723483503aecc8512cfec2a937989e89450c7ab4eb26e1b5c1d6 WatchSource:0}: Error finding container 275e6fcc8013723483503aecc8512cfec2a937989e89450c7ab4eb26e1b5c1d6: Status 404 returned error can't find the container with id 275e6fcc8013723483503aecc8512cfec2a937989e89450c7ab4eb26e1b5c1d6
Nov 02 23:01:17.638994 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:17.638890    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"]
Nov 02 23:01:18.538491 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:18.538458    4528 generic.go:356] "Generic (PLEG): container finished" podID="5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e" containerID="014d2d07dcf01ccf198de2222271ac6cafcf6dc2f9a05e90e781b8867925ea64" exitCode=0
Nov 02 23:01:18.538896 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:18.538506    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk" event={"ID":"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e","Type":"ContainerDied","Data":"014d2d07dcf01ccf198de2222271ac6cafcf6dc2f9a05e90e781b8867925ea64"}
Nov 02 23:01:18.538896 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:18.538560    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk" event={"ID":"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e","Type":"ContainerStarted","Data":"275e6fcc8013723483503aecc8512cfec2a937989e89450c7ab4eb26e1b5c1d6"}
Nov 02 23:01:19.674547 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.674525    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:19.823394 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.823327    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-openshift-ca\") pod \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") "
Nov 02 23:01:19.823539 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.823418    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-kube-api-access-swtp2\" (UniqueName: \"kubernetes.io/projected/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-kube-api-access-swtp2\") pod \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") "
Nov 02 23:01:19.823539 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.823464    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-xks-ca\") pod \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\" (UID: \"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e\") "
Nov 02 23:01:19.823729 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.823709    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-openshift-ca" pod "5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e" (UID: "5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:01:19.823792 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.823780    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:19.825551 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.825527    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-xks-ca" pod "5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e" (UID: "5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:01:19.825627 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.825604    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-kube-api-access-swtp2" pod "5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e" (UID: "5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e"). InnerVolumeSpecName "kube-api-access-swtp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:01:19.925268 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.925239    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swtp2\" (UniqueName: \"kubernetes.io/projected/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-kube-api-access-swtp2\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:19.925268 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:19.925264    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:20.549912 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:20.549877    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk" event={"ID":"5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e","Type":"ContainerDied","Data":"275e6fcc8013723483503aecc8512cfec2a937989e89450c7ab4eb26e1b5c1d6"}
Nov 02 23:01:20.550104 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:20.549914    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162e8e3-nrmbk"
Nov 02 23:01:20.550104 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:20.549933    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="275e6fcc8013723483503aecc8512cfec2a937989e89450c7ab4eb26e1b5c1d6"
Nov 02 23:01:27.862902 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.862869    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"]
Nov 02 23:01:27.863316 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.863242    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e" containerName="revoke-keys"
Nov 02 23:01:27.863316 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.863254    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e" containerName="revoke-keys"
Nov 02 23:01:27.863391 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.863325    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e" containerName="revoke-keys"
Nov 02 23:01:27.866449 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.866432    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:27.918453 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.918425    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"]
Nov 02 23:01:27.966541 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.966510    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:01:27.966541 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.966526    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:01:27.966541 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.966527    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:01:27.968605 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.968592    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:01:27.996993 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.996967    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2q4\" (UniqueName: \"kubernetes.io/projected/62328619-84b0-4820-9d67-e35c3544b970-kube-api-access-dl2q4\") pod \"maas-api-revoke-sub-980024ae-q6stk\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:27.997144 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.997009    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62328619-84b0-4820-9d67-e35c3544b970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-980024ae-q6stk\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:27.997144 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:27.997095    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62328619-84b0-4820-9d67-e35c3544b970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-980024ae-q6stk\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:28.097890 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:28.097859    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2q4\" (UniqueName: \"kubernetes.io/projected/62328619-84b0-4820-9d67-e35c3544b970-kube-api-access-dl2q4\") pod \"maas-api-revoke-sub-980024ae-q6stk\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:28.098042 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:28.097910    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62328619-84b0-4820-9d67-e35c3544b970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-980024ae-q6stk\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:28.098042 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:28.097948    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62328619-84b0-4820-9d67-e35c3544b970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-980024ae-q6stk\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:28.098639 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:28.098616    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62328619-84b0-4820-9d67-e35c3544b970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-980024ae-q6stk\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:28.100247 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:28.100228    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62328619-84b0-4820-9d67-e35c3544b970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-980024ae-q6stk\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:28.155656 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:28.155601    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2q4\" (UniqueName: \"kubernetes.io/projected/62328619-84b0-4820-9d67-e35c3544b970-kube-api-access-dl2q4\") pod \"maas-api-revoke-sub-980024ae-q6stk\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:28.183446 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:28.183422    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:28.547084 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:01:28.547053    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62328619_84b0_4820_9d67_e35c3544b970.slice/crio-5646ca910bef89d093328b64618201c19d24cf7af727971c2eda4c56f83a3965 WatchSource:0}: Error finding container 5646ca910bef89d093328b64618201c19d24cf7af727971c2eda4c56f83a3965: Status 404 returned error can't find the container with id 5646ca910bef89d093328b64618201c19d24cf7af727971c2eda4c56f83a3965
Nov 02 23:01:28.548696 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:28.548582    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"]
Nov 02 23:01:28.591040 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:28.591012    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk" event={"ID":"62328619-84b0-4820-9d67-e35c3544b970","Type":"ContainerStarted","Data":"5646ca910bef89d093328b64618201c19d24cf7af727971c2eda4c56f83a3965"}
Nov 02 23:01:29.597827 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:29.597794    4528 generic.go:356] "Generic (PLEG): container finished" podID="62328619-84b0-4820-9d67-e35c3544b970" containerID="2d63f740b917a8dbd2ed3ff005310bf9ac1197798afd319a922787cd9a876315" exitCode=0
Nov 02 23:01:29.598234 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:29.597885    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk" event={"ID":"62328619-84b0-4820-9d67-e35c3544b970","Type":"ContainerDied","Data":"2d63f740b917a8dbd2ed3ff005310bf9ac1197798afd319a922787cd9a876315"}
Nov 02 23:01:30.747803 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:30.747781    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:30.925145 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:30.925063    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62328619-84b0-4820-9d67-e35c3544b970-kube-api-access-dl2q4\" (UniqueName: \"kubernetes.io/projected/62328619-84b0-4820-9d67-e35c3544b970-kube-api-access-dl2q4\") pod \"62328619-84b0-4820-9d67-e35c3544b970\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") "
Nov 02 23:01:30.925276 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:30.925247    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62328619-84b0-4820-9d67-e35c3544b970-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62328619-84b0-4820-9d67-e35c3544b970-maas-api-xks-ca\") pod \"62328619-84b0-4820-9d67-e35c3544b970\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") "
Nov 02 23:01:30.925314 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:30.925282    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62328619-84b0-4820-9d67-e35c3544b970-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62328619-84b0-4820-9d67-e35c3544b970-maas-api-openshift-ca\") pod \"62328619-84b0-4820-9d67-e35c3544b970\" (UID: \"62328619-84b0-4820-9d67-e35c3544b970\") "
Nov 02 23:01:30.925659 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:30.925632    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62328619-84b0-4820-9d67-e35c3544b970-maas-api-openshift-ca" pod "62328619-84b0-4820-9d67-e35c3544b970" (UID: "62328619-84b0-4820-9d67-e35c3544b970"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:01:30.927136 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:30.927095    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62328619-84b0-4820-9d67-e35c3544b970-maas-api-xks-ca" pod "62328619-84b0-4820-9d67-e35c3544b970" (UID: "62328619-84b0-4820-9d67-e35c3544b970"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:01:30.927251 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:30.927229    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62328619-84b0-4820-9d67-e35c3544b970-kube-api-access-dl2q4" pod "62328619-84b0-4820-9d67-e35c3544b970" (UID: "62328619-84b0-4820-9d67-e35c3544b970"). InnerVolumeSpecName "kube-api-access-dl2q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:01:31.026921 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:31.026890    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62328619-84b0-4820-9d67-e35c3544b970-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:31.026921 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:31.026922    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62328619-84b0-4820-9d67-e35c3544b970-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:31.027083 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:31.026937    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl2q4\" (UniqueName: \"kubernetes.io/projected/62328619-84b0-4820-9d67-e35c3544b970-kube-api-access-dl2q4\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:31.448460 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:31.448427    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 23:01:31.608495 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:31.608459    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk" event={"ID":"62328619-84b0-4820-9d67-e35c3544b970","Type":"ContainerDied","Data":"5646ca910bef89d093328b64618201c19d24cf7af727971c2eda4c56f83a3965"}
Nov 02 23:01:31.608495 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:31.608480    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980024ae-q6stk"
Nov 02 23:01:31.608751 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:31.608506    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5646ca910bef89d093328b64618201c19d24cf7af727971c2eda4c56f83a3965"
Nov 02 23:01:38.532701 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.532667    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"]
Nov 02 23:01:38.533209 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.533050    4528 cpu_manager.go:414] "containerMap: removing container" podUID="62328619-84b0-4820-9d67-e35c3544b970" containerName="revoke-keys"
Nov 02 23:01:38.533209 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.533062    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62328619-84b0-4820-9d67-e35c3544b970" containerName="revoke-keys"
Nov 02 23:01:38.533209 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.533143    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="62328619-84b0-4820-9d67-e35c3544b970" containerName="revoke-keys"
Nov 02 23:01:38.536451 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.536432    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.589837 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.589804    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"]
Nov 02 23:01:38.625419 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.625390    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc2c2bd-zh588\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.625557 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.625522    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc2c2bd-zh588\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.625557 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.625551    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxnlv\" (UniqueName: \"kubernetes.io/projected/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-kube-api-access-zxnlv\") pod \"maas-api-revoke-sub-2cc2c2bd-zh588\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.637839 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.637792    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:01:38.637903 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.637839    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:01:38.639083 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.639062    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:01:38.639083 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.639076    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:01:38.726611 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.726579    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc2c2bd-zh588\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.726611 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.726614    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxnlv\" (UniqueName: \"kubernetes.io/projected/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-kube-api-access-zxnlv\") pod \"maas-api-revoke-sub-2cc2c2bd-zh588\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.726823 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.726648    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc2c2bd-zh588\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.727354 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.727332    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc2c2bd-zh588\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.728920 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.728886    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc2c2bd-zh588\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.783994 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.783973    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxnlv\" (UniqueName: \"kubernetes.io/projected/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-kube-api-access-zxnlv\") pod \"maas-api-revoke-sub-2cc2c2bd-zh588\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:38.849555 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:38.849528    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:39.212720 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:01:39.212688    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9caced5e_0a0a_4d77_a8bb_65d4ea2a0978.slice/crio-52c037c4671ddb3928f0bd6ae2a63112f7040ea610d9c80ffccdf708308e9809 WatchSource:0}: Error finding container 52c037c4671ddb3928f0bd6ae2a63112f7040ea610d9c80ffccdf708308e9809: Status 404 returned error can't find the container with id 52c037c4671ddb3928f0bd6ae2a63112f7040ea610d9c80ffccdf708308e9809
Nov 02 23:01:39.214649 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:39.214625    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"]
Nov 02 23:01:39.679768 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:39.679730    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588" event={"ID":"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978","Type":"ContainerStarted","Data":"418f504bdae1cce5052e32ebcfea253ca3e1a17a3a814422c54531d95f45a108"}
Nov 02 23:01:39.680192 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:39.679830    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588" event={"ID":"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978","Type":"ContainerStarted","Data":"52c037c4671ddb3928f0bd6ae2a63112f7040ea610d9c80ffccdf708308e9809"}
Nov 02 23:01:39.795177 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:39.795084    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588" podStartSLOduration=1.7950661079999999 podStartE2EDuration="1.795066108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:39.792621021 +0000 UTC m=+6787.056157338" watchObservedRunningTime="2026-11-02 23:01:39.795066108 +0000 UTC m=+6787.058602429"
Nov 02 23:01:40.686050 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:40.686014    4528 generic.go:356] "Generic (PLEG): container finished" podID="9caced5e-0a0a-4d77-a8bb-65d4ea2a0978" containerID="418f504bdae1cce5052e32ebcfea253ca3e1a17a3a814422c54531d95f45a108" exitCode=0
Nov 02 23:01:40.686459 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:40.686100    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588" event={"ID":"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978","Type":"ContainerDied","Data":"418f504bdae1cce5052e32ebcfea253ca3e1a17a3a814422c54531d95f45a108"}
Nov 02 23:01:41.818396 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.818372    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:41.855779 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.855749    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-openshift-ca\") pod \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") "
Nov 02 23:01:41.855935 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.855808    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-kube-api-access-zxnlv\" (UniqueName: \"kubernetes.io/projected/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-kube-api-access-zxnlv\") pod \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") "
Nov 02 23:01:41.855935 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.855862    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-xks-ca\") pod \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\" (UID: \"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978\") "
Nov 02 23:01:41.856096 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.856064    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-openshift-ca" pod "9caced5e-0a0a-4d77-a8bb-65d4ea2a0978" (UID: "9caced5e-0a0a-4d77-a8bb-65d4ea2a0978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:01:41.856242 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.856209    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:41.857758 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.857741    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-xks-ca" pod "9caced5e-0a0a-4d77-a8bb-65d4ea2a0978" (UID: "9caced5e-0a0a-4d77-a8bb-65d4ea2a0978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:01:41.857855 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.857839    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-kube-api-access-zxnlv" pod "9caced5e-0a0a-4d77-a8bb-65d4ea2a0978" (UID: "9caced5e-0a0a-4d77-a8bb-65d4ea2a0978"). InnerVolumeSpecName "kube-api-access-zxnlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:01:41.957020 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.956940    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxnlv\" (UniqueName: \"kubernetes.io/projected/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-kube-api-access-zxnlv\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:41.957020 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:41.956971    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caced5e-0a0a-4d77-a8bb-65d4ea2a0978-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:42.697262 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:42.697230    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588"
Nov 02 23:01:42.697449 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:42.697236    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc2c2bd-zh588" event={"ID":"9caced5e-0a0a-4d77-a8bb-65d4ea2a0978","Type":"ContainerDied","Data":"52c037c4671ddb3928f0bd6ae2a63112f7040ea610d9c80ffccdf708308e9809"}
Nov 02 23:01:42.697449 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:42.697343    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c037c4671ddb3928f0bd6ae2a63112f7040ea610d9c80ffccdf708308e9809"
Nov 02 23:01:44.111644 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:01:44.111572    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:01:49.747792 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.747759    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"]
Nov 02 23:01:49.748196 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.748135    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9caced5e-0a0a-4d77-a8bb-65d4ea2a0978" containerName="revoke-keys"
Nov 02 23:01:49.748196 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.748148    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9caced5e-0a0a-4d77-a8bb-65d4ea2a0978" containerName="revoke-keys"
Nov 02 23:01:49.748278 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.748225    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9caced5e-0a0a-4d77-a8bb-65d4ea2a0978" containerName="revoke-keys"
Nov 02 23:01:49.751497 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.751482    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:49.804269 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.804237    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"]
Nov 02 23:01:49.828180 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.828147    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7be5185d-zwfcc\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:49.828371 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.828352    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7be5185d-zwfcc\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:49.828438 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.828389    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9z9w\" (UniqueName: \"kubernetes.io/projected/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-kube-api-access-f9z9w\") pod \"maas-api-revoke-sub-7be5185d-zwfcc\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:49.851010 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.850979    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:01:49.851145 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.851006    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:01:49.851145 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.851017    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:01:49.853945 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.853930    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:01:49.929182 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.929153    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7be5185d-zwfcc\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:49.929182 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.929184    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9z9w\" (UniqueName: \"kubernetes.io/projected/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-kube-api-access-f9z9w\") pod \"maas-api-revoke-sub-7be5185d-zwfcc\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:49.929373 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.929240    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7be5185d-zwfcc\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:49.929821 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.929801    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7be5185d-zwfcc\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:49.931591 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.931567    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7be5185d-zwfcc\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:49.985788 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:49.985763    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9z9w\" (UniqueName: \"kubernetes.io/projected/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-kube-api-access-f9z9w\") pod \"maas-api-revoke-sub-7be5185d-zwfcc\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:50.063880 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:50.063853    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:50.427704 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:01:50.427668    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfa322f_5f4e_48c0_bdac_8dc8b6676022.slice/crio-9d2936b88429db10971ea500fe8be83ffe87fa3c6f9b5ef55148de9ff589a3ba WatchSource:0}: Error finding container 9d2936b88429db10971ea500fe8be83ffe87fa3c6f9b5ef55148de9ff589a3ba: Status 404 returned error can't find the container with id 9d2936b88429db10971ea500fe8be83ffe87fa3c6f9b5ef55148de9ff589a3ba
Nov 02 23:01:50.430373 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:50.430231    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"]
Nov 02 23:01:50.736035 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:50.735965    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc" event={"ID":"3cfa322f-5f4e-48c0-bdac-8dc8b6676022","Type":"ContainerStarted","Data":"f30403f5618140d7193754b148f628f70fb367db6a354eff4da5e5bea52282a4"}
Nov 02 23:01:50.736035 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:50.736015    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc" event={"ID":"3cfa322f-5f4e-48c0-bdac-8dc8b6676022","Type":"ContainerStarted","Data":"9d2936b88429db10971ea500fe8be83ffe87fa3c6f9b5ef55148de9ff589a3ba"}
Nov 02 23:01:50.850308 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:50.850257    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc" podStartSLOduration=1.850244024 podStartE2EDuration="1.850244024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:50.848025673 +0000 UTC m=+6798.111561972" watchObservedRunningTime="2026-11-02 23:01:50.850244024 +0000 UTC m=+6798.113780341"
Nov 02 23:01:51.742046 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:51.742013    4528 generic.go:356] "Generic (PLEG): container finished" podID="3cfa322f-5f4e-48c0-bdac-8dc8b6676022" containerID="f30403f5618140d7193754b148f628f70fb367db6a354eff4da5e5bea52282a4" exitCode=0
Nov 02 23:01:51.742240 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:51.742091    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc" event={"ID":"3cfa322f-5f4e-48c0-bdac-8dc8b6676022","Type":"ContainerDied","Data":"f30403f5618140d7193754b148f628f70fb367db6a354eff4da5e5bea52282a4"}
Nov 02 23:01:52.884783 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:52.884758    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:52.957530 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:52.957501    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-openshift-ca\") pod \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") "
Nov 02 23:01:52.957717 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:52.957575    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-kube-api-access-f9z9w\" (UniqueName: \"kubernetes.io/projected/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-kube-api-access-f9z9w\") pod \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") "
Nov 02 23:01:52.957792 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:52.957732    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-xks-ca\") pod \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\" (UID: \"3cfa322f-5f4e-48c0-bdac-8dc8b6676022\") "
Nov 02 23:01:52.957911 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:52.957886    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-openshift-ca" pod "3cfa322f-5f4e-48c0-bdac-8dc8b6676022" (UID: "3cfa322f-5f4e-48c0-bdac-8dc8b6676022"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:01:52.958153 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:52.958101    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:52.959629 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:52.959602    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-xks-ca" pod "3cfa322f-5f4e-48c0-bdac-8dc8b6676022" (UID: "3cfa322f-5f4e-48c0-bdac-8dc8b6676022"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:01:52.959748 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:52.959727    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-kube-api-access-f9z9w" pod "3cfa322f-5f4e-48c0-bdac-8dc8b6676022" (UID: "3cfa322f-5f4e-48c0-bdac-8dc8b6676022"). InnerVolumeSpecName "kube-api-access-f9z9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:01:53.058825 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:53.058792    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9z9w\" (UniqueName: \"kubernetes.io/projected/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-kube-api-access-f9z9w\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:53.058825 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:53.058821    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfa322f-5f4e-48c0-bdac-8dc8b6676022-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:53.752533 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:53.752501    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc"
Nov 02 23:01:53.752716 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:53.752505    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7be5185d-zwfcc" event={"ID":"3cfa322f-5f4e-48c0-bdac-8dc8b6676022","Type":"ContainerDied","Data":"9d2936b88429db10971ea500fe8be83ffe87fa3c6f9b5ef55148de9ff589a3ba"}
Nov 02 23:01:53.752716 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:53.752644    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d2936b88429db10971ea500fe8be83ffe87fa3c6f9b5ef55148de9ff589a3ba"
Nov 02 23:01:55.842753 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:55.842705    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tgp7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:01:55.843247 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:55.842799    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tgp7n" podUID="5ddfda63-2274-4793-bb2a-e99d52000804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:01:55.856541 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:55.856509    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gj9qp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:01:55.856698 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:01:55.856577    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gj9qp" podUID="2a0bbf59-4728-4a9a-882a-44eec7d799df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:02:00.426706 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.426667    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"]
Nov 02 23:02:00.427123 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.427057    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3cfa322f-5f4e-48c0-bdac-8dc8b6676022" containerName="revoke-keys"
Nov 02 23:02:00.427123 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.427069    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfa322f-5f4e-48c0-bdac-8dc8b6676022" containerName="revoke-keys"
Nov 02 23:02:00.427209 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.427156    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfa322f-5f4e-48c0-bdac-8dc8b6676022" containerName="revoke-keys"
Nov 02 23:02:00.430028 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.430013    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.484310 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.484276    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"]
Nov 02 23:02:00.527533 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.527501    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cd77fb4-c9zxj\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.527678 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.527563    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97gb\" (UniqueName: \"kubernetes.io/projected/8fed627b-8b3e-4254-b15f-671d3fbdf34f-kube-api-access-t97gb\") pod \"maas-api-revoke-sub-2cd77fb4-c9zxj\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.527678 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.527642    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cd77fb4-c9zxj\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.531248 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.531226    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:02:00.531248 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.531241    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:02:00.533395 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.533370    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:02:00.533488 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.533439    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:02:00.629120 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.629080    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cd77fb4-c9zxj\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.629318 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.629176    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97gb\" (UniqueName: \"kubernetes.io/projected/8fed627b-8b3e-4254-b15f-671d3fbdf34f-kube-api-access-t97gb\") pod \"maas-api-revoke-sub-2cd77fb4-c9zxj\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.629318 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.629206    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cd77fb4-c9zxj\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.629685 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.629664    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cd77fb4-c9zxj\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.631547 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.631525    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cd77fb4-c9zxj\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.686057 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.685988    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97gb\" (UniqueName: \"kubernetes.io/projected/8fed627b-8b3e-4254-b15f-671d3fbdf34f-kube-api-access-t97gb\") pod \"maas-api-revoke-sub-2cd77fb4-c9zxj\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:00.742043 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:00.742009    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:01.105585 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:02:01.105555    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fed627b_8b3e_4254_b15f_671d3fbdf34f.slice/crio-b15cbaa17f55f27e14571d69a71ce0d7ced73b53fc4486b30be2fd3dcbcd4259 WatchSource:0}: Error finding container b15cbaa17f55f27e14571d69a71ce0d7ced73b53fc4486b30be2fd3dcbcd4259: Status 404 returned error can't find the container with id b15cbaa17f55f27e14571d69a71ce0d7ced73b53fc4486b30be2fd3dcbcd4259
Nov 02 23:02:01.107948 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:01.107922    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"]
Nov 02 23:02:01.815451 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:01.815415    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj" event={"ID":"8fed627b-8b3e-4254-b15f-671d3fbdf34f","Type":"ContainerStarted","Data":"a3e14fa7d59448570a667200a203f09aff652bf776723d429585d3d0eb4ea5b7"}
Nov 02 23:02:01.815856 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:01.815475    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj" event={"ID":"8fed627b-8b3e-4254-b15f-671d3fbdf34f","Type":"ContainerStarted","Data":"b15cbaa17f55f27e14571d69a71ce0d7ced73b53fc4486b30be2fd3dcbcd4259"}
Nov 02 23:02:01.929517 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:01.929466    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj" podStartSLOduration=1.929451127 podStartE2EDuration="1.929451127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:01.927812097 +0000 UTC m=+6809.191348413" watchObservedRunningTime="2026-11-02 23:02:01.929451127 +0000 UTC m=+6809.192987440"
Nov 02 23:02:02.822541 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:02.822502    4528 generic.go:356] "Generic (PLEG): container finished" podID="8fed627b-8b3e-4254-b15f-671d3fbdf34f" containerID="a3e14fa7d59448570a667200a203f09aff652bf776723d429585d3d0eb4ea5b7" exitCode=0
Nov 02 23:02:02.822947 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:02.822580    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj" event={"ID":"8fed627b-8b3e-4254-b15f-671d3fbdf34f","Type":"ContainerDied","Data":"a3e14fa7d59448570a667200a203f09aff652bf776723d429585d3d0eb4ea5b7"}
Nov 02 23:02:03.952766 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:03.952740    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:04.063512 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.063464    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-openshift-ca\") pod \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") "
Nov 02 23:02:04.063674 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.063638    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fed627b-8b3e-4254-b15f-671d3fbdf34f-kube-api-access-t97gb\" (UniqueName: \"kubernetes.io/projected/8fed627b-8b3e-4254-b15f-671d3fbdf34f-kube-api-access-t97gb\") pod \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") "
Nov 02 23:02:04.063773 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.063671    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-xks-ca\") pod \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\" (UID: \"8fed627b-8b3e-4254-b15f-671d3fbdf34f\") "
Nov 02 23:02:04.063982 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.063951    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-openshift-ca" pod "8fed627b-8b3e-4254-b15f-671d3fbdf34f" (UID: "8fed627b-8b3e-4254-b15f-671d3fbdf34f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:02:04.066055 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.066027    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fed627b-8b3e-4254-b15f-671d3fbdf34f-kube-api-access-t97gb" pod "8fed627b-8b3e-4254-b15f-671d3fbdf34f" (UID: "8fed627b-8b3e-4254-b15f-671d3fbdf34f"). InnerVolumeSpecName "kube-api-access-t97gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:02:04.066180 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.066096    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-xks-ca" pod "8fed627b-8b3e-4254-b15f-671d3fbdf34f" (UID: "8fed627b-8b3e-4254-b15f-671d3fbdf34f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:02:04.164798 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.164699    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t97gb\" (UniqueName: \"kubernetes.io/projected/8fed627b-8b3e-4254-b15f-671d3fbdf34f-kube-api-access-t97gb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:04.164798 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.164745    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:04.164798 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.164754    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed627b-8b3e-4254-b15f-671d3fbdf34f-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:04.833459 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.833422    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj" event={"ID":"8fed627b-8b3e-4254-b15f-671d3fbdf34f","Type":"ContainerDied","Data":"b15cbaa17f55f27e14571d69a71ce0d7ced73b53fc4486b30be2fd3dcbcd4259"}
Nov 02 23:02:04.833459 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.833460    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd77fb4-c9zxj"
Nov 02 23:02:04.833459 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:04.833470    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b15cbaa17f55f27e14571d69a71ce0d7ced73b53fc4486b30be2fd3dcbcd4259"
Nov 02 23:02:11.599969 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.599927    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"]
Nov 02 23:02:11.600576 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.600552    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8fed627b-8b3e-4254-b15f-671d3fbdf34f" containerName="revoke-keys"
Nov 02 23:02:11.600624 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.600582    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fed627b-8b3e-4254-b15f-671d3fbdf34f" containerName="revoke-keys"
Nov 02 23:02:11.600748 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.600733    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fed627b-8b3e-4254-b15f-671d3fbdf34f" containerName="revoke-keys"
Nov 02 23:02:11.604141 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.604094    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.654515 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.654479    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"]
Nov 02 23:02:11.701847 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.701817    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:02:11.701847 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.701845    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:02:11.702067 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.701848    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:02:11.702067 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.701863    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:02:11.732552 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.732516    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d7d6\" (UniqueName: \"kubernetes.io/projected/51d931d8-f918-4507-b635-cf17bd490d06-kube-api-access-2d7d6\") pod \"maas-api-revoke-sub-9c8ca465-twlcs\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.732735 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.732635    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c8ca465-twlcs\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.732735 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.732689    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c8ca465-twlcs\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.833707 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.833678    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d7d6\" (UniqueName: \"kubernetes.io/projected/51d931d8-f918-4507-b635-cf17bd490d06-kube-api-access-2d7d6\") pod \"maas-api-revoke-sub-9c8ca465-twlcs\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.833857 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.833743    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c8ca465-twlcs\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.833857 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.833764    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c8ca465-twlcs\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.834405 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.834385    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c8ca465-twlcs\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.835996 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.835978    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c8ca465-twlcs\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.891412 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.891340    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d7d6\" (UniqueName: \"kubernetes.io/projected/51d931d8-f918-4507-b635-cf17bd490d06-kube-api-access-2d7d6\") pod \"maas-api-revoke-sub-9c8ca465-twlcs\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:11.917239 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:11.917216    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:12.280055 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:02:12.280024    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d931d8_f918_4507_b635_cf17bd490d06.slice/crio-d2ed48d441afe05cdb50ec07b38ea6187068af0b1d5e4241276e58e595afe2aa WatchSource:0}: Error finding container d2ed48d441afe05cdb50ec07b38ea6187068af0b1d5e4241276e58e595afe2aa: Status 404 returned error can't find the container with id d2ed48d441afe05cdb50ec07b38ea6187068af0b1d5e4241276e58e595afe2aa
Nov 02 23:02:12.281890 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:12.281867    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"]
Nov 02 23:02:12.872185 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:12.872088    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs" event={"ID":"51d931d8-f918-4507-b635-cf17bd490d06","Type":"ContainerStarted","Data":"c70ae354d84dd31bb920240495d7a2f3e393a92dac8354b81e33a107b3442546"}
Nov 02 23:02:12.872185 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:12.872151    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs" event={"ID":"51d931d8-f918-4507-b635-cf17bd490d06","Type":"ContainerStarted","Data":"d2ed48d441afe05cdb50ec07b38ea6187068af0b1d5e4241276e58e595afe2aa"}
Nov 02 23:02:12.985499 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:12.985447    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs" podStartSLOduration=1.98543268 podStartE2EDuration="1.98543268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:12.985188363 +0000 UTC m=+6820.248724680" watchObservedRunningTime="2026-11-02 23:02:12.98543268 +0000 UTC m=+6820.248968997"
Nov 02 23:02:13.878916 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:13.878885    4528 generic.go:356] "Generic (PLEG): container finished" podID="51d931d8-f918-4507-b635-cf17bd490d06" containerID="c70ae354d84dd31bb920240495d7a2f3e393a92dac8354b81e33a107b3442546" exitCode=0
Nov 02 23:02:13.879422 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:13.878980    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs" event={"ID":"51d931d8-f918-4507-b635-cf17bd490d06","Type":"ContainerDied","Data":"c70ae354d84dd31bb920240495d7a2f3e393a92dac8354b81e33a107b3442546"}
Nov 02 23:02:14.113311 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:02:14.113274    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:02:15.015031 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.015009    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:15.163990 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.163897    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-xks-ca\") pod \"51d931d8-f918-4507-b635-cf17bd490d06\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") "
Nov 02 23:02:15.163990 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.163933    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-openshift-ca\") pod \"51d931d8-f918-4507-b635-cf17bd490d06\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") "
Nov 02 23:02:15.164241 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.163998    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d931d8-f918-4507-b635-cf17bd490d06-kube-api-access-2d7d6\" (UniqueName: \"kubernetes.io/projected/51d931d8-f918-4507-b635-cf17bd490d06-kube-api-access-2d7d6\") pod \"51d931d8-f918-4507-b635-cf17bd490d06\" (UID: \"51d931d8-f918-4507-b635-cf17bd490d06\") "
Nov 02 23:02:15.164376 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.164353    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-openshift-ca" pod "51d931d8-f918-4507-b635-cf17bd490d06" (UID: "51d931d8-f918-4507-b635-cf17bd490d06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:02:15.166152 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.166084    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-xks-ca" pod "51d931d8-f918-4507-b635-cf17bd490d06" (UID: "51d931d8-f918-4507-b635-cf17bd490d06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:02:15.166152 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.166147    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d931d8-f918-4507-b635-cf17bd490d06-kube-api-access-2d7d6" pod "51d931d8-f918-4507-b635-cf17bd490d06" (UID: "51d931d8-f918-4507-b635-cf17bd490d06"). InnerVolumeSpecName "kube-api-access-2d7d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:02:15.265543 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.265512    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:15.265543 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.265539    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d931d8-f918-4507-b635-cf17bd490d06-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:15.265543 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.265548    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d7d6\" (UniqueName: \"kubernetes.io/projected/51d931d8-f918-4507-b635-cf17bd490d06-kube-api-access-2d7d6\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:15.891411 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.891373    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs" event={"ID":"51d931d8-f918-4507-b635-cf17bd490d06","Type":"ContainerDied","Data":"d2ed48d441afe05cdb50ec07b38ea6187068af0b1d5e4241276e58e595afe2aa"}
Nov 02 23:02:15.891607 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.891443    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ed48d441afe05cdb50ec07b38ea6187068af0b1d5e4241276e58e595afe2aa"
Nov 02 23:02:15.891607 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:15.891384    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c8ca465-twlcs"
Nov 02 23:02:22.287834 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.287792    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"]
Nov 02 23:02:22.288384 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.288368    4528 cpu_manager.go:414] "containerMap: removing container" podUID="51d931d8-f918-4507-b635-cf17bd490d06" containerName="revoke-keys"
Nov 02 23:02:22.288457 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.288388    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d931d8-f918-4507-b635-cf17bd490d06" containerName="revoke-keys"
Nov 02 23:02:22.288519 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.288491    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d931d8-f918-4507-b635-cf17bd490d06" containerName="revoke-keys"
Nov 02 23:02:22.291801 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.291779    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.334733 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.334683    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46235d54-9759-4207-82f6-605d32e01f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baba9d0b-scjlp\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.334941 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.334758    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46235d54-9759-4207-82f6-605d32e01f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baba9d0b-scjlp\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.334941 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.334826    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdsl\" (UniqueName: \"kubernetes.io/projected/46235d54-9759-4207-82f6-605d32e01f11-kube-api-access-ftdsl\") pod \"maas-api-revoke-sub-baba9d0b-scjlp\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.342882 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.342851    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"]
Nov 02 23:02:22.390767 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.390733    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:02:22.390947 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.390852    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:02:22.394468 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.394449    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:02:22.394541 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.394461    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:02:22.436257 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.436224    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46235d54-9759-4207-82f6-605d32e01f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baba9d0b-scjlp\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.436408 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.436267    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdsl\" (UniqueName: \"kubernetes.io/projected/46235d54-9759-4207-82f6-605d32e01f11-kube-api-access-ftdsl\") pod \"maas-api-revoke-sub-baba9d0b-scjlp\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.436408 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.436323    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46235d54-9759-4207-82f6-605d32e01f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baba9d0b-scjlp\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.436894 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.436866    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46235d54-9759-4207-82f6-605d32e01f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baba9d0b-scjlp\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.438588 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.438570    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46235d54-9759-4207-82f6-605d32e01f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baba9d0b-scjlp\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.497805 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.497770    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdsl\" (UniqueName: \"kubernetes.io/projected/46235d54-9759-4207-82f6-605d32e01f11-kube-api-access-ftdsl\") pod \"maas-api-revoke-sub-baba9d0b-scjlp\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.604395 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.604310    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:22.968635 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:02:22.968601    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46235d54_9759_4207_82f6_605d32e01f11.slice/crio-5b0d8d7cd666a5a2e5575000ed1ad2c65cc6ff00f2b636c1e6e7d88ba06c57c4 WatchSource:0}: Error finding container 5b0d8d7cd666a5a2e5575000ed1ad2c65cc6ff00f2b636c1e6e7d88ba06c57c4: Status 404 returned error can't find the container with id 5b0d8d7cd666a5a2e5575000ed1ad2c65cc6ff00f2b636c1e6e7d88ba06c57c4
Nov 02 23:02:22.971499 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:22.971477    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"]
Nov 02 23:02:23.931009 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:23.930975    4528 generic.go:356] "Generic (PLEG): container finished" podID="46235d54-9759-4207-82f6-605d32e01f11" containerID="c2b26975921fd8ba49b358cb4cbc7082c92ab0f149a27e7015561511f4f624c8" exitCode=0
Nov 02 23:02:23.931434 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:23.931064    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp" event={"ID":"46235d54-9759-4207-82f6-605d32e01f11","Type":"ContainerDied","Data":"c2b26975921fd8ba49b358cb4cbc7082c92ab0f149a27e7015561511f4f624c8"}
Nov 02 23:02:23.931434 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:23.931168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp" event={"ID":"46235d54-9759-4207-82f6-605d32e01f11","Type":"ContainerStarted","Data":"5b0d8d7cd666a5a2e5575000ed1ad2c65cc6ff00f2b636c1e6e7d88ba06c57c4"}
Nov 02 23:02:25.073503 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.073477    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:25.159638 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.159603    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46235d54-9759-4207-82f6-605d32e01f11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46235d54-9759-4207-82f6-605d32e01f11-maas-api-xks-ca\") pod \"46235d54-9759-4207-82f6-605d32e01f11\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") "
Nov 02 23:02:25.159810 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.159673    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46235d54-9759-4207-82f6-605d32e01f11-kube-api-access-ftdsl\" (UniqueName: \"kubernetes.io/projected/46235d54-9759-4207-82f6-605d32e01f11-kube-api-access-ftdsl\") pod \"46235d54-9759-4207-82f6-605d32e01f11\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") "
Nov 02 23:02:25.159810 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.159787    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46235d54-9759-4207-82f6-605d32e01f11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46235d54-9759-4207-82f6-605d32e01f11-maas-api-openshift-ca\") pod \"46235d54-9759-4207-82f6-605d32e01f11\" (UID: \"46235d54-9759-4207-82f6-605d32e01f11\") "
Nov 02 23:02:25.160164 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.160142    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46235d54-9759-4207-82f6-605d32e01f11-maas-api-openshift-ca" pod "46235d54-9759-4207-82f6-605d32e01f11" (UID: "46235d54-9759-4207-82f6-605d32e01f11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:02:25.161813 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.161792    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46235d54-9759-4207-82f6-605d32e01f11-maas-api-xks-ca" pod "46235d54-9759-4207-82f6-605d32e01f11" (UID: "46235d54-9759-4207-82f6-605d32e01f11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:02:25.161898 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.161807    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46235d54-9759-4207-82f6-605d32e01f11-kube-api-access-ftdsl" pod "46235d54-9759-4207-82f6-605d32e01f11" (UID: "46235d54-9759-4207-82f6-605d32e01f11"). InnerVolumeSpecName "kube-api-access-ftdsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:02:25.261374 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.261345    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46235d54-9759-4207-82f6-605d32e01f11-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:25.261374 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.261374    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46235d54-9759-4207-82f6-605d32e01f11-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:25.261571 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.261385    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftdsl\" (UniqueName: \"kubernetes.io/projected/46235d54-9759-4207-82f6-605d32e01f11-kube-api-access-ftdsl\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:25.942038 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.942010    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp"
Nov 02 23:02:25.942221 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.942057    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baba9d0b-scjlp" event={"ID":"46235d54-9759-4207-82f6-605d32e01f11","Type":"ContainerDied","Data":"5b0d8d7cd666a5a2e5575000ed1ad2c65cc6ff00f2b636c1e6e7d88ba06c57c4"}
Nov 02 23:02:25.942221 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:25.942127    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b0d8d7cd666a5a2e5575000ed1ad2c65cc6ff00f2b636c1e6e7d88ba06c57c4"
Nov 02 23:02:33.194363 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.194326    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"]
Nov 02 23:02:33.194805 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.194737    4528 cpu_manager.go:414] "containerMap: removing container" podUID="46235d54-9759-4207-82f6-605d32e01f11" containerName="revoke-keys"
Nov 02 23:02:33.194805 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.194749    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46235d54-9759-4207-82f6-605d32e01f11" containerName="revoke-keys"
Nov 02 23:02:33.194893 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.194811    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="46235d54-9759-4207-82f6-605d32e01f11" containerName="revoke-keys"
Nov 02 23:02:33.197911 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.197890    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.249942 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.249903    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"]
Nov 02 23:02:33.297711 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.297679    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:02:33.297919 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.297782    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:02:33.300677 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.300660    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:02:33.300677 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.300654    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:02:33.342513 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.342477    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92b63d61-h8zcl\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.342709 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.342605    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92b63d61-h8zcl\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.342709 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.342686    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjl7g\" (UniqueName: \"kubernetes.io/projected/35c8ef57-7951-47ee-942b-ec89eab0219a-kube-api-access-xjl7g\") pod \"maas-api-revoke-sub-92b63d61-h8zcl\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.444030 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.443989    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92b63d61-h8zcl\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.444258 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.444073    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjl7g\" (UniqueName: \"kubernetes.io/projected/35c8ef57-7951-47ee-942b-ec89eab0219a-kube-api-access-xjl7g\") pod \"maas-api-revoke-sub-92b63d61-h8zcl\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.444258 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.444103    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92b63d61-h8zcl\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.444727 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.444684    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92b63d61-h8zcl\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.446462 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.446444    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92b63d61-h8zcl\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.501540 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.501506    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjl7g\" (UniqueName: \"kubernetes.io/projected/35c8ef57-7951-47ee-942b-ec89eab0219a-kube-api-access-xjl7g\") pod \"maas-api-revoke-sub-92b63d61-h8zcl\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.513342 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.513310    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:33.878738 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:02:33.878704    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c8ef57_7951_47ee_942b_ec89eab0219a.slice/crio-49dad5e3e1ebf923084b9d8ee6ca9ea33603ec0938676108d3e88481a2dd49da WatchSource:0}: Error finding container 49dad5e3e1ebf923084b9d8ee6ca9ea33603ec0938676108d3e88481a2dd49da: Status 404 returned error can't find the container with id 49dad5e3e1ebf923084b9d8ee6ca9ea33603ec0938676108d3e88481a2dd49da
Nov 02 23:02:33.880942 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.880904    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"]
Nov 02 23:02:33.980452 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:33.980417    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl" event={"ID":"35c8ef57-7951-47ee-942b-ec89eab0219a","Type":"ContainerStarted","Data":"49dad5e3e1ebf923084b9d8ee6ca9ea33603ec0938676108d3e88481a2dd49da"}
Nov 02 23:02:34.986798 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:34.986704    4528 generic.go:356] "Generic (PLEG): container finished" podID="35c8ef57-7951-47ee-942b-ec89eab0219a" containerID="516dd260201491de07d8ba95d4c3a44b2e13bcfffe6dfd2cefbdeacefcdea4c7" exitCode=0
Nov 02 23:02:34.987208 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:34.986799    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl" event={"ID":"35c8ef57-7951-47ee-942b-ec89eab0219a","Type":"ContainerDied","Data":"516dd260201491de07d8ba95d4c3a44b2e13bcfffe6dfd2cefbdeacefcdea4c7"}
Nov 02 23:02:36.166459 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.166429    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:36.273123 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.273062    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-xks-ca\") pod \"35c8ef57-7951-47ee-942b-ec89eab0219a\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") "
Nov 02 23:02:36.273393 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.273160    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-openshift-ca\") pod \"35c8ef57-7951-47ee-942b-ec89eab0219a\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") "
Nov 02 23:02:36.273393 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.273195    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c8ef57-7951-47ee-942b-ec89eab0219a-kube-api-access-xjl7g\" (UniqueName: \"kubernetes.io/projected/35c8ef57-7951-47ee-942b-ec89eab0219a-kube-api-access-xjl7g\") pod \"35c8ef57-7951-47ee-942b-ec89eab0219a\" (UID: \"35c8ef57-7951-47ee-942b-ec89eab0219a\") "
Nov 02 23:02:36.274025 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.273978    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-openshift-ca" pod "35c8ef57-7951-47ee-942b-ec89eab0219a" (UID: "35c8ef57-7951-47ee-942b-ec89eab0219a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:02:36.302838 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.302531    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c8ef57-7951-47ee-942b-ec89eab0219a-kube-api-access-xjl7g" pod "35c8ef57-7951-47ee-942b-ec89eab0219a" (UID: "35c8ef57-7951-47ee-942b-ec89eab0219a"). InnerVolumeSpecName "kube-api-access-xjl7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:02:36.319484 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.319417    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-xks-ca" pod "35c8ef57-7951-47ee-942b-ec89eab0219a" (UID: "35c8ef57-7951-47ee-942b-ec89eab0219a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:02:36.375096 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.375008    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:36.375096 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.375047    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c8ef57-7951-47ee-942b-ec89eab0219a-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:36.375096 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.375064    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjl7g\" (UniqueName: \"kubernetes.io/projected/35c8ef57-7951-47ee-942b-ec89eab0219a-kube-api-access-xjl7g\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:36.999643 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.999605    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl" event={"ID":"35c8ef57-7951-47ee-942b-ec89eab0219a","Type":"ContainerDied","Data":"49dad5e3e1ebf923084b9d8ee6ca9ea33603ec0938676108d3e88481a2dd49da"}
Nov 02 23:02:36.999943 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:36.999926    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49dad5e3e1ebf923084b9d8ee6ca9ea33603ec0938676108d3e88481a2dd49da"
Nov 02 23:02:37.000162 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:37.000147    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92b63d61-h8zcl"
Nov 02 23:02:37.338248 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:37.338205    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-pwm9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:02:37.338828 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:37.338799    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-pwm9l" podUID="00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:02:43.931925 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:43.931887    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"]
Nov 02 23:02:43.932483 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:43.932292    4528 cpu_manager.go:414] "containerMap: removing container" podUID="35c8ef57-7951-47ee-942b-ec89eab0219a" containerName="revoke-keys"
Nov 02 23:02:43.932483 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:43.932304    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c8ef57-7951-47ee-942b-ec89eab0219a" containerName="revoke-keys"
Nov 02 23:02:43.932483 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:43.932370    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c8ef57-7951-47ee-942b-ec89eab0219a" containerName="revoke-keys"
Nov 02 23:02:43.935713 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:43.935697    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:43.987719 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:43.987679    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"]
Nov 02 23:02:44.035533 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.035501    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:02:44.035533 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.035523    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:02:44.035769 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.035597    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:02:44.038414 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.038395    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:02:44.042776 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.042755    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e62531b4-4x4xc\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.042847 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.042808    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e62531b4-4x4xc\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.042925 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.042910    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5lbb\" (UniqueName: \"kubernetes.io/projected/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-kube-api-access-s5lbb\") pod \"maas-api-revoke-sub-e62531b4-4x4xc\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.113173 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:02:44.113134    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:02:44.143516 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.143482    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lbb\" (UniqueName: \"kubernetes.io/projected/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-kube-api-access-s5lbb\") pod \"maas-api-revoke-sub-e62531b4-4x4xc\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.143707 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.143545    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e62531b4-4x4xc\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.143707 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.143586    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e62531b4-4x4xc\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.144214 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.144193    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e62531b4-4x4xc\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.145967 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.145939    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e62531b4-4x4xc\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.201066 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.200995    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5lbb\" (UniqueName: \"kubernetes.io/projected/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-kube-api-access-s5lbb\") pod \"maas-api-revoke-sub-e62531b4-4x4xc\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.248662 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.248629    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:44.616462 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:02:44.616401    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d2bd8c_aebe_4ec9_8a4f_c62a3e829f56.slice/crio-ba340e734c2b9a834fa28624f846305de724d43e904c2c1489853ec2326758ec WatchSource:0}: Error finding container ba340e734c2b9a834fa28624f846305de724d43e904c2c1489853ec2326758ec: Status 404 returned error can't find the container with id ba340e734c2b9a834fa28624f846305de724d43e904c2c1489853ec2326758ec
Nov 02 23:02:44.619334 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:44.619305    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"]
Nov 02 23:02:45.042367 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:45.042334    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc" event={"ID":"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56","Type":"ContainerStarted","Data":"0ba5aecf7a8cc94e1c33422b5abb9f73bd390643fc79da74ff91259c51927719"}
Nov 02 23:02:45.042789 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:45.042387    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc" event={"ID":"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56","Type":"ContainerStarted","Data":"ba340e734c2b9a834fa28624f846305de724d43e904c2c1489853ec2326758ec"}
Nov 02 23:02:45.157796 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:45.157742    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc" podStartSLOduration=2.157729198 podStartE2EDuration="2.157729198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:45.155195865 +0000 UTC m=+6852.418732178" watchObservedRunningTime="2026-11-02 23:02:45.157729198 +0000 UTC m=+6852.421265493"
Nov 02 23:02:46.048498 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:46.048463    4528 generic.go:356] "Generic (PLEG): container finished" podID="25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56" containerID="0ba5aecf7a8cc94e1c33422b5abb9f73bd390643fc79da74ff91259c51927719" exitCode=0
Nov 02 23:02:46.048952 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:46.048557    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc" event={"ID":"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56","Type":"ContainerDied","Data":"0ba5aecf7a8cc94e1c33422b5abb9f73bd390643fc79da74ff91259c51927719"}
Nov 02 23:02:47.341986 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.341340    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:47.394792 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.394756    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-kube-api-access-s5lbb\" (UniqueName: \"kubernetes.io/projected/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-kube-api-access-s5lbb\") pod \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") "
Nov 02 23:02:47.394995 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.394839    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-xks-ca\") pod \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") "
Nov 02 23:02:47.394995 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.394904    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-openshift-ca\") pod \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\" (UID: \"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56\") "
Nov 02 23:02:47.395312 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.395289    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-openshift-ca" pod "25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56" (UID: "25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:02:47.396950 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.396919    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-xks-ca" pod "25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56" (UID: "25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:02:47.397062 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.396985    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-kube-api-access-s5lbb" pod "25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56" (UID: "25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56"). InnerVolumeSpecName "kube-api-access-s5lbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:02:47.496523 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.496488    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:47.496523 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.496517    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5lbb\" (UniqueName: \"kubernetes.io/projected/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-kube-api-access-s5lbb\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:47.496523 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:47.496528    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:48.077035 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:48.076988    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc" event={"ID":"25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56","Type":"ContainerDied","Data":"ba340e734c2b9a834fa28624f846305de724d43e904c2c1489853ec2326758ec"}
Nov 02 23:02:48.077293 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:48.077060    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba340e734c2b9a834fa28624f846305de724d43e904c2c1489853ec2326758ec"
Nov 02 23:02:48.077293 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:48.077167    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62531b4-4x4xc"
Nov 02 23:02:54.671406 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.671372    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"]
Nov 02 23:02:54.671817 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.671749    4528 cpu_manager.go:414] "containerMap: removing container" podUID="25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56" containerName="revoke-keys"
Nov 02 23:02:54.671817 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.671759    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56" containerName="revoke-keys"
Nov 02 23:02:54.671889 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.671838    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56" containerName="revoke-keys"
Nov 02 23:02:54.681010 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.677700    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.727077 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.727044    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"]
Nov 02 23:02:54.758694 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.758664    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eebeac83-vf5br\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.758878 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.758700    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv242\" (UniqueName: \"kubernetes.io/projected/4dc3742d-cc96-4c9f-942b-c11cc57da317-kube-api-access-gv242\") pod \"maas-api-revoke-sub-eebeac83-vf5br\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.758878 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.758848    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eebeac83-vf5br\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.774672 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.774640    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:02:54.774672 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.774664    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:02:54.774831 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.774700    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:02:54.780521 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.780507    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:02:54.859824 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.859790    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eebeac83-vf5br\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.860043 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.859846    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eebeac83-vf5br\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.860043 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.859960    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv242\" (UniqueName: \"kubernetes.io/projected/4dc3742d-cc96-4c9f-942b-c11cc57da317-kube-api-access-gv242\") pod \"maas-api-revoke-sub-eebeac83-vf5br\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.860558 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.860536    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eebeac83-vf5br\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.862245 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.862228    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eebeac83-vf5br\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.916433 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.916403    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv242\" (UniqueName: \"kubernetes.io/projected/4dc3742d-cc96-4c9f-942b-c11cc57da317-kube-api-access-gv242\") pod \"maas-api-revoke-sub-eebeac83-vf5br\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:54.992599 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:54.992511    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:55.365694 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:02:55.365357    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc3742d_cc96_4c9f_942b_c11cc57da317.slice/crio-aaf1fdd64e09f7c81905309d05b84812745480c92a382c39729072561b4d1ee5 WatchSource:0}: Error finding container aaf1fdd64e09f7c81905309d05b84812745480c92a382c39729072561b4d1ee5: Status 404 returned error can't find the container with id aaf1fdd64e09f7c81905309d05b84812745480c92a382c39729072561b4d1ee5
Nov 02 23:02:55.365694 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:55.365484    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"]
Nov 02 23:02:56.118070 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:56.118040    4528 generic.go:356] "Generic (PLEG): container finished" podID="4dc3742d-cc96-4c9f-942b-c11cc57da317" containerID="0974dad892d563258b5fd6254ea795a206c06c50e6cc0ca259add4869d0fced5" exitCode=0
Nov 02 23:02:56.118462 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:56.118132    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br" event={"ID":"4dc3742d-cc96-4c9f-942b-c11cc57da317","Type":"ContainerDied","Data":"0974dad892d563258b5fd6254ea795a206c06c50e6cc0ca259add4869d0fced5"}
Nov 02 23:02:56.118462 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:56.118189    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br" event={"ID":"4dc3742d-cc96-4c9f-942b-c11cc57da317","Type":"ContainerStarted","Data":"aaf1fdd64e09f7c81905309d05b84812745480c92a382c39729072561b4d1ee5"}
Nov 02 23:02:57.281611 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.281585    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:57.385405 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.385297    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-openshift-ca\") pod \"4dc3742d-cc96-4c9f-942b-c11cc57da317\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") "
Nov 02 23:02:57.385580 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.385409    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-xks-ca\") pod \"4dc3742d-cc96-4c9f-942b-c11cc57da317\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") "
Nov 02 23:02:57.385580 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.385450    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc3742d-cc96-4c9f-942b-c11cc57da317-kube-api-access-gv242\" (UniqueName: \"kubernetes.io/projected/4dc3742d-cc96-4c9f-942b-c11cc57da317-kube-api-access-gv242\") pod \"4dc3742d-cc96-4c9f-942b-c11cc57da317\" (UID: \"4dc3742d-cc96-4c9f-942b-c11cc57da317\") "
Nov 02 23:02:57.386147 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.386101    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-openshift-ca" pod "4dc3742d-cc96-4c9f-942b-c11cc57da317" (UID: "4dc3742d-cc96-4c9f-942b-c11cc57da317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:02:57.388640 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.388537    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-xks-ca" pod "4dc3742d-cc96-4c9f-942b-c11cc57da317" (UID: "4dc3742d-cc96-4c9f-942b-c11cc57da317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:02:57.388640 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.388582    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc3742d-cc96-4c9f-942b-c11cc57da317-kube-api-access-gv242" pod "4dc3742d-cc96-4c9f-942b-c11cc57da317" (UID: "4dc3742d-cc96-4c9f-942b-c11cc57da317"). InnerVolumeSpecName "kube-api-access-gv242". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:02:57.486999 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.486952    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:57.486999 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.486993    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc3742d-cc96-4c9f-942b-c11cc57da317-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:57.487340 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:57.487011    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv242\" (UniqueName: \"kubernetes.io/projected/4dc3742d-cc96-4c9f-942b-c11cc57da317-kube-api-access-gv242\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:58.049770 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:58.049712    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"]
Nov 02 23:02:58.098175 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:58.098139    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cd3b896-bwjhx"]
Nov 02 23:02:58.169133 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:58.157142    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br" event={"ID":"4dc3742d-cc96-4c9f-942b-c11cc57da317","Type":"ContainerDied","Data":"aaf1fdd64e09f7c81905309d05b84812745480c92a382c39729072561b4d1ee5"}
Nov 02 23:02:58.169942 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:58.169912    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf1fdd64e09f7c81905309d05b84812745480c92a382c39729072561b4d1ee5"
Nov 02 23:02:58.170201 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:58.170183    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eebeac83-vf5br"
Nov 02 23:02:58.935919 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:02:58.935881    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa90522a-9b40-40dd-8fe8-f5a48161fe4f" path="/var/lib/kubelet/pods/aa90522a-9b40-40dd-8fe8-f5a48161fe4f/volumes"
Nov 02 23:03:05.461798 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.461768    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"]
Nov 02 23:03:05.462290 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.462134    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4dc3742d-cc96-4c9f-942b-c11cc57da317" containerName="revoke-keys"
Nov 02 23:03:05.462290 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.462144    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc3742d-cc96-4c9f-942b-c11cc57da317" containerName="revoke-keys"
Nov 02 23:03:05.462290 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.462220    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc3742d-cc96-4c9f-942b-c11cc57da317" containerName="revoke-keys"
Nov 02 23:03:05.465222 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.465205    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.516917 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.516869    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"]
Nov 02 23:03:05.564837 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.564802    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:03:05.564837 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.564802    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:03:05.564837 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.564808    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:03:05.567740 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.567724    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:03:05.586152 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.586088    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee7d98d4-lnf8j\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.586332 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.586214    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njw8f\" (UniqueName: \"kubernetes.io/projected/20ca7b08-150d-406d-a6a5-4b8a54936af5-kube-api-access-njw8f\") pod \"maas-api-revoke-sub-ee7d98d4-lnf8j\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.586332 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.586316    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee7d98d4-lnf8j\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.687433 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.687391    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee7d98d4-lnf8j\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.687433 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.687442    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee7d98d4-lnf8j\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.687728 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.687561    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njw8f\" (UniqueName: \"kubernetes.io/projected/20ca7b08-150d-406d-a6a5-4b8a54936af5-kube-api-access-njw8f\") pod \"maas-api-revoke-sub-ee7d98d4-lnf8j\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.688209 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.688187    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee7d98d4-lnf8j\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.689782 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.689761    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee7d98d4-lnf8j\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.744670 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.744593    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njw8f\" (UniqueName: \"kubernetes.io/projected/20ca7b08-150d-406d-a6a5-4b8a54936af5-kube-api-access-njw8f\") pod \"maas-api-revoke-sub-ee7d98d4-lnf8j\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:05.778323 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:05.778287    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:06.143874 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:03:06.143839    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ca7b08_150d_406d_a6a5_4b8a54936af5.slice/crio-e47c40f40288a67867393fd7e022324567f18c6eb7844b4675a00c5f9bf64dfd WatchSource:0}: Error finding container e47c40f40288a67867393fd7e022324567f18c6eb7844b4675a00c5f9bf64dfd: Status 404 returned error can't find the container with id e47c40f40288a67867393fd7e022324567f18c6eb7844b4675a00c5f9bf64dfd
Nov 02 23:03:06.146358 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:06.146331    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"]
Nov 02 23:03:06.217156 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:06.217098    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j" event={"ID":"20ca7b08-150d-406d-a6a5-4b8a54936af5","Type":"ContainerStarted","Data":"e47c40f40288a67867393fd7e022324567f18c6eb7844b4675a00c5f9bf64dfd"}
Nov 02 23:03:07.224015 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:07.223934    4528 generic.go:356] "Generic (PLEG): container finished" podID="20ca7b08-150d-406d-a6a5-4b8a54936af5" containerID="4d32512ea853ee255dc2341629e083d03db7f56c651c1f065edb4666601d6e77" exitCode=0
Nov 02 23:03:07.224411 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:07.224018    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j" event={"ID":"20ca7b08-150d-406d-a6a5-4b8a54936af5","Type":"ContainerDied","Data":"4d32512ea853ee255dc2341629e083d03db7f56c651c1f065edb4666601d6e77"}
Nov 02 23:03:08.048518 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.048466    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"]
Nov 02 23:03:08.098771 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.098728    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10877a1e-rgkvk"]
Nov 02 23:03:08.411556 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.411523    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:08.515205 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.515165    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-xks-ca\") pod \"20ca7b08-150d-406d-a6a5-4b8a54936af5\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") "
Nov 02 23:03:08.515417 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.515248    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-openshift-ca\") pod \"20ca7b08-150d-406d-a6a5-4b8a54936af5\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") "
Nov 02 23:03:08.515417 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.515316    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ca7b08-150d-406d-a6a5-4b8a54936af5-kube-api-access-njw8f\" (UniqueName: \"kubernetes.io/projected/20ca7b08-150d-406d-a6a5-4b8a54936af5-kube-api-access-njw8f\") pod \"20ca7b08-150d-406d-a6a5-4b8a54936af5\" (UID: \"20ca7b08-150d-406d-a6a5-4b8a54936af5\") "
Nov 02 23:03:08.515851 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.515823    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-openshift-ca" pod "20ca7b08-150d-406d-a6a5-4b8a54936af5" (UID: "20ca7b08-150d-406d-a6a5-4b8a54936af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:03:08.517765 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.517744    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-xks-ca" pod "20ca7b08-150d-406d-a6a5-4b8a54936af5" (UID: "20ca7b08-150d-406d-a6a5-4b8a54936af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:03:08.517980 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.517962    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ca7b08-150d-406d-a6a5-4b8a54936af5-kube-api-access-njw8f" pod "20ca7b08-150d-406d-a6a5-4b8a54936af5" (UID: "20ca7b08-150d-406d-a6a5-4b8a54936af5"). InnerVolumeSpecName "kube-api-access-njw8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:03:08.616734 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.616573    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:03:08.616734 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.616613    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njw8f\" (UniqueName: \"kubernetes.io/projected/20ca7b08-150d-406d-a6a5-4b8a54936af5-kube-api-access-njw8f\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:03:08.616734 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.616627    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ca7b08-150d-406d-a6a5-4b8a54936af5-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:03:08.940183 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:08.939820    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25f77e8-8271-447b-b224-77aabd7b1d1d" path="/var/lib/kubelet/pods/b25f77e8-8271-447b-b224-77aabd7b1d1d/volumes"
Nov 02 23:03:09.274185 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:09.263706    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j" event={"ID":"20ca7b08-150d-406d-a6a5-4b8a54936af5","Type":"ContainerDied","Data":"e47c40f40288a67867393fd7e022324567f18c6eb7844b4675a00c5f9bf64dfd"}
Nov 02 23:03:09.274185 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:09.263789    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e47c40f40288a67867393fd7e022324567f18c6eb7844b4675a00c5f9bf64dfd"
Nov 02 23:03:09.274185 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:09.263876    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7d98d4-lnf8j"
Nov 02 23:03:14.110865 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:03:14.110825    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:03:16.202636 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.202604    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"]
Nov 02 23:03:16.203150 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.202987    4528 cpu_manager.go:414] "containerMap: removing container" podUID="20ca7b08-150d-406d-a6a5-4b8a54936af5" containerName="revoke-keys"
Nov 02 23:03:16.203150 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.202999    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ca7b08-150d-406d-a6a5-4b8a54936af5" containerName="revoke-keys"
Nov 02 23:03:16.203150 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.203073    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ca7b08-150d-406d-a6a5-4b8a54936af5" containerName="revoke-keys"
Nov 02 23:03:16.207225 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.207201    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.227893 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.227868    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dvm\" (UniqueName: \"kubernetes.io/projected/f3728368-2fbd-4d04-b75b-299d58b4d3f4-kube-api-access-n5dvm\") pod \"maas-api-revoke-sub-8afb869b-2zbfb\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.228024 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.227901    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8afb869b-2zbfb\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.228139 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.228100    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8afb869b-2zbfb\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.258437 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.258410    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"]
Nov 02 23:03:16.306147 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.306100    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 23:03:16.306147 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.306135    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 23:03:16.309661 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.309636    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 23:03:16.309770 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.309717    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\""
Nov 02 23:03:16.328748 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.328723    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8afb869b-2zbfb\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.328871 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.328757    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dvm\" (UniqueName: \"kubernetes.io/projected/f3728368-2fbd-4d04-b75b-299d58b4d3f4-kube-api-access-n5dvm\") pod \"maas-api-revoke-sub-8afb869b-2zbfb\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.328871 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.328775    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8afb869b-2zbfb\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.329357 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.329340    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8afb869b-2zbfb\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.331134 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.331102    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8afb869b-2zbfb\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.385134 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.385096    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dvm\" (UniqueName: \"kubernetes.io/projected/f3728368-2fbd-4d04-b75b-299d58b4d3f4-kube-api-access-n5dvm\") pod \"maas-api-revoke-sub-8afb869b-2zbfb\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.520316 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.520291    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:16.889480 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:03:16.889434    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3728368_2fbd_4d04_b75b_299d58b4d3f4.slice/crio-75292646bf340b984803567ef1bd9355d0830ff1d231632b3f8cb03b5f920bb8 WatchSource:0}: Error finding container 75292646bf340b984803567ef1bd9355d0830ff1d231632b3f8cb03b5f920bb8: Status 404 returned error can't find the container with id 75292646bf340b984803567ef1bd9355d0830ff1d231632b3f8cb03b5f920bb8
Nov 02 23:03:16.891520 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:16.891487    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"]
Nov 02 23:03:17.309856 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:17.309818    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb" event={"ID":"f3728368-2fbd-4d04-b75b-299d58b4d3f4","Type":"ContainerStarted","Data":"51e084d9b1ef0d69562f975d7fcc3fccf08c8cb243637c2ccba81f574813a14d"}
Nov 02 23:03:17.310261 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:17.309868    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb" event={"ID":"f3728368-2fbd-4d04-b75b-299d58b4d3f4","Type":"ContainerStarted","Data":"75292646bf340b984803567ef1bd9355d0830ff1d231632b3f8cb03b5f920bb8"}
Nov 02 23:03:17.427752 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:17.427700    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb" podStartSLOduration=1.427687537 podStartE2EDuration="1.427687537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:17.425857077 +0000 UTC m=+6884.689393393" watchObservedRunningTime="2026-11-02 23:03:17.427687537 +0000 UTC m=+6884.691223853"
Nov 02 23:03:18.318303 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:18.318260    4528 generic.go:356] "Generic (PLEG): container finished" podID="f3728368-2fbd-4d04-b75b-299d58b4d3f4" containerID="51e084d9b1ef0d69562f975d7fcc3fccf08c8cb243637c2ccba81f574813a14d" exitCode=0
Nov 02 23:03:18.318857 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:18.318343    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb" event={"ID":"f3728368-2fbd-4d04-b75b-299d58b4d3f4","Type":"ContainerDied","Data":"51e084d9b1ef0d69562f975d7fcc3fccf08c8cb243637c2ccba81f574813a14d"}
Nov 02 23:03:19.698050 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.698026    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:19.777574 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.777536    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-xks-ca\") pod \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") "
Nov 02 23:03:19.777890 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.777870    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3728368-2fbd-4d04-b75b-299d58b4d3f4-kube-api-access-n5dvm\" (UniqueName: \"kubernetes.io/projected/f3728368-2fbd-4d04-b75b-299d58b4d3f4-kube-api-access-n5dvm\") pod \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") "
Nov 02 23:03:19.778471 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.778449    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-openshift-ca\") pod \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\" (UID: \"f3728368-2fbd-4d04-b75b-299d58b4d3f4\") "
Nov 02 23:03:19.779315 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.778924    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-openshift-ca" pod "f3728368-2fbd-4d04-b75b-299d58b4d3f4" (UID: "f3728368-2fbd-4d04-b75b-299d58b4d3f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:03:19.780973 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.780951    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3728368-2fbd-4d04-b75b-299d58b4d3f4-kube-api-access-n5dvm" pod "f3728368-2fbd-4d04-b75b-299d58b4d3f4" (UID: "f3728368-2fbd-4d04-b75b-299d58b4d3f4"). InnerVolumeSpecName "kube-api-access-n5dvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:03:19.781911 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.781892    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-xks-ca" pod "f3728368-2fbd-4d04-b75b-299d58b4d3f4" (UID: "f3728368-2fbd-4d04-b75b-299d58b4d3f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:03:19.880629 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.880591    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-xks-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:03:19.880879 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.880858    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5dvm\" (UniqueName: \"kubernetes.io/projected/f3728368-2fbd-4d04-b75b-299d58b4d3f4-kube-api-access-n5dvm\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:03:19.881021 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:19.881004    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3728368-2fbd-4d04-b75b-299d58b4d3f4-maas-api-openshift-ca\") on node \"ip-10-0-133-86.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:03:20.350591 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:20.350553    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb" event={"ID":"f3728368-2fbd-4d04-b75b-299d58b4d3f4","Type":"ContainerDied","Data":"75292646bf340b984803567ef1bd9355d0830ff1d231632b3f8cb03b5f920bb8"}
Nov 02 23:03:20.350870 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:20.350849    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75292646bf340b984803567ef1bd9355d0830ff1d231632b3f8cb03b5f920bb8"
Nov 02 23:03:20.351060 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:20.351044    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afb869b-2zbfb"
Nov 02 23:03:28.869229 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:28.869194    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5tp5v"]
Nov 02 23:03:41.045762 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:41.045717    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"]
Nov 02 23:03:41.095867 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:41.095830    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ceea6ff-gjzj8"]
Nov 02 23:03:42.912037 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:42.912000    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bce206b-b859-485f-865f-502ea6b24edb" path="/var/lib/kubelet/pods/1bce206b-b859-485f-865f-502ea6b24edb/volumes"
Nov 02 23:03:43.227840 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:43.227764    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867cc8b678-sgzth_774878d1-3662-465a-8a44-3fa542eefd66/authorino/0.log"
Nov 02 23:03:44.118626 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:03:44.118586    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:03:54.476294 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:54.476266    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74879c7667-jgdxh_8ddc5d0a-c7a0-412a-ac84-c522dfcba78c/manager/0.log"
Nov 02 23:03:54.829297 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:54.829265    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9xjsn_4be9c715-baba-45d6-8cd9-19d679e10c2a/manager/0.log"
Nov 02 23:03:56.340566 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:56.340536    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rgnpj_cb32ed85-0c0b-4b06-95c0-8cf300e337f5/server/0.log"
Nov 02 23:03:58.328090 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:58.328060    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2cc2c2bd-zh588_9caced5e-0a0a-4d77-a8bb-65d4ea2a0978/revoke-keys/0.log"
Nov 02 23:03:58.537709 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:58.537678    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2cd77fb4-c9zxj_8fed627b-8b3e-4254-b15f-671d3fbdf34f/revoke-keys/0.log"
Nov 02 23:03:58.738338 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:58.738265    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3ab714c1-s8hwr_d3daecb0-ca6c-4b4c-aac4-ef17a9417a1a/revoke-keys/0.log"
Nov 02 23:03:58.944476 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:58.944434    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-43286820-8hgzf_708ccd0d-118a-40fb-ba51-9ddc657cb131/revoke-keys/0.log"
Nov 02 23:03:59.156088 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:59.156060    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7be5185d-zwfcc_3cfa322f-5f4e-48c0-bdac-8dc8b6676022/revoke-keys/0.log"
Nov 02 23:03:59.366463 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:59.366439    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8afb869b-2zbfb_f3728368-2fbd-4d04-b75b-299d58b4d3f4/revoke-keys/0.log"
Nov 02 23:03:59.578697 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:59.578665    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-92b63d61-h8zcl_35c8ef57-7951-47ee-942b-ec89eab0219a/revoke-keys/0.log"
Nov 02 23:03:59.775454 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:59.775424    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-980024ae-q6stk_62328619-84b0-4820-9d67-e35c3544b970/revoke-keys/0.log"
Nov 02 23:03:59.974481 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:03:59.974402    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9c8ca465-twlcs_51d931d8-f918-4507-b635-cf17bd490d06/revoke-keys/0.log"
Nov 02 23:04:00.171347 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:00.171316    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a162e8e3-nrmbk_5e8d95d4-fa9d-4d3c-ab27-03daefc5f78e/revoke-keys/0.log"
Nov 02 23:04:00.373566 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:00.373538    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b79f698c-chn77_4dcba5b8-a32a-41d7-96fb-6161baea6fea/revoke-keys/0.log"
Nov 02 23:04:00.583157 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:00.583132    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-baba9d0b-scjlp_46235d54-9759-4207-82f6-605d32e01f11/revoke-keys/0.log"
Nov 02 23:04:00.786952 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:00.786917    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e26f2aba-46ck9_57d07466-7e06-4094-987e-a9ac0d9d3723/revoke-keys/0.log"
Nov 02 23:04:00.993486 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:00.993449    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e62531b4-4x4xc_25d2bd8c-aebe-4ec9-8a4f-c62a3e829f56/revoke-keys/0.log"
Nov 02 23:04:01.206859 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:01.206783    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ee7d98d4-lnf8j_20ca7b08-150d-406d-a6a5-4b8a54936af5/revoke-keys/0.log"
Nov 02 23:04:01.414811 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:01.414785    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eebeac83-vf5br_4dc3742d-cc96-4c9f-942b-c11cc57da317/revoke-keys/0.log"
Nov 02 23:04:03.454969 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:03.454929    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867cc8b678-sgzth_774878d1-3662-465a-8a44-3fa542eefd66/authorino/0.log"
Nov 02 23:04:05.069959 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:05.069934    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5tp5v_598f64cf-7fb5-4b62-a4ee-29802c83c4f4/limitador/0.log"
Nov 02 23:04:06.442948 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:06.442921    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-n5c5h_c167e9c0-18fb-447d-b5ce-febde46c229f/discovery/0.log"
Nov 02 23:04:06.948936 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:06.948903    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78fbf7b575-zlh7p_cb4c26f6-5512-4b9d-9c66-0a2a45798eb3/kube-auth-proxy/0.log"
Nov 02 23:04:08.230341 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:08.230305    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-gj9qp_2a0bbf59-4728-4a9a-882a-44eec7d799df/istio-proxy/0.log"
Nov 02 23:04:08.955066 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:08.955025    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-pwm9l_00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd/istio-proxy/0.log"
Nov 02 23:04:09.173395 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:09.173368    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-tgp7n_5ddfda63-2274-4793-bb2a-e99d52000804/istio-proxy/0.log"
Nov 02 23:04:09.889846 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:09.889817    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7669ccf7ff-28hfp_cc1bf636-d125-4f0c-b76f-6a11c1f26457/payload-pre-processing/0.log"
Nov 02 23:04:10.095534 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:10.095510    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6cd6f7444d-c7gdn_d0d9de36-b748-4d05-9e97-2182dbed6f6b/payload-processing/0.log"
Nov 02 23:04:12.036446 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:12.036405    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-zgd28_31358659-6618-4da0-90bd-1d69aefa7ffd/main/0.log"
Nov 02 23:04:12.247746 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:12.247715    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-rmwm5_bf97d442-ef72-4399-a586-88274dfbd464/main/0.log"
Nov 02 23:04:12.665688 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:12.665662    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-mqnnt_dec65c14-0c66-48d0-98ad-c190b3ed0e38/main/0.log"
Nov 02 23:04:12.862502 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:12.862472    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-dstck_92fa4184-1b2b-49ca-8517-06957cad7927/main/0.log"
Nov 02 23:04:14.112218 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:04:14.112181    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:04:22.433195 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.433157    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vlpkd/must-gather-ng6vh"]
Nov 02 23:04:22.433811 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.433726    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f3728368-2fbd-4d04-b75b-299d58b4d3f4" containerName="revoke-keys"
Nov 02 23:04:22.433811 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.433746    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3728368-2fbd-4d04-b75b-299d58b4d3f4" containerName="revoke-keys"
Nov 02 23:04:22.433947 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.433856    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3728368-2fbd-4d04-b75b-299d58b4d3f4" containerName="revoke-keys"
Nov 02 23:04:22.437243 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.437224    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlpkd/must-gather-ng6vh"
Nov 02 23:04:22.481056 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.481031    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xt5s\" (UniqueName: \"kubernetes.io/projected/a4ee7a8f-b582-44c8-9460-3e0466bcb3bd-kube-api-access-8xt5s\") pod \"must-gather-ng6vh\" (UID: \"a4ee7a8f-b582-44c8-9460-3e0466bcb3bd\") " pod="openshift-must-gather-vlpkd/must-gather-ng6vh"
Nov 02 23:04:22.481203 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.481184    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4ee7a8f-b582-44c8-9460-3e0466bcb3bd-must-gather-output\") pod \"must-gather-ng6vh\" (UID: \"a4ee7a8f-b582-44c8-9460-3e0466bcb3bd\") " pod="openshift-must-gather-vlpkd/must-gather-ng6vh"
Nov 02 23:04:22.498458 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.498429    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlpkd/must-gather-ng6vh"]
Nov 02 23:04:22.544664 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.544644    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlpkd\"/\"openshift-service-ca.crt\""
Nov 02 23:04:22.544801 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.544704    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlpkd\"/\"kube-root-ca.crt\""
Nov 02 23:04:22.544801 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.544746    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vlpkd\"/\"default-dockercfg-v24v2\""
Nov 02 23:04:22.581695 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.581658    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4ee7a8f-b582-44c8-9460-3e0466bcb3bd-must-gather-output\") pod \"must-gather-ng6vh\" (UID: \"a4ee7a8f-b582-44c8-9460-3e0466bcb3bd\") " pod="openshift-must-gather-vlpkd/must-gather-ng6vh"
Nov 02 23:04:22.581695 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.581699    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xt5s\" (UniqueName: \"kubernetes.io/projected/a4ee7a8f-b582-44c8-9460-3e0466bcb3bd-kube-api-access-8xt5s\") pod \"must-gather-ng6vh\" (UID: \"a4ee7a8f-b582-44c8-9460-3e0466bcb3bd\") " pod="openshift-must-gather-vlpkd/must-gather-ng6vh"
Nov 02 23:04:22.582028 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.582006    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4ee7a8f-b582-44c8-9460-3e0466bcb3bd-must-gather-output\") pod \"must-gather-ng6vh\" (UID: \"a4ee7a8f-b582-44c8-9460-3e0466bcb3bd\") " pod="openshift-must-gather-vlpkd/must-gather-ng6vh"
Nov 02 23:04:22.639192 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.639167    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xt5s\" (UniqueName: \"kubernetes.io/projected/a4ee7a8f-b582-44c8-9460-3e0466bcb3bd-kube-api-access-8xt5s\") pod \"must-gather-ng6vh\" (UID: \"a4ee7a8f-b582-44c8-9460-3e0466bcb3bd\") " pod="openshift-must-gather-vlpkd/must-gather-ng6vh"
Nov 02 23:04:22.750762 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:22.750682    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlpkd/must-gather-ng6vh"
Nov 02 23:04:23.116147 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:04:23.116104    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4ee7a8f_b582_44c8_9460_3e0466bcb3bd.slice/crio-1ca72998e7a1910f642dadf2476e88fc23692758ac3d8ea3db17ee57923c2f4a WatchSource:0}: Error finding container 1ca72998e7a1910f642dadf2476e88fc23692758ac3d8ea3db17ee57923c2f4a: Status 404 returned error can't find the container with id 1ca72998e7a1910f642dadf2476e88fc23692758ac3d8ea3db17ee57923c2f4a
Nov 02 23:04:23.118257 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:23.118236    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 23:04:23.118371 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:23.118349    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlpkd/must-gather-ng6vh"]
Nov 02 23:04:23.676134 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:23.676082    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlpkd/must-gather-ng6vh" event={"ID":"a4ee7a8f-b582-44c8-9460-3e0466bcb3bd","Type":"ContainerStarted","Data":"1ca72998e7a1910f642dadf2476e88fc23692758ac3d8ea3db17ee57923c2f4a"}
Nov 02 23:04:24.046913 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:24.046872    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"]
Nov 02 23:04:24.097385 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:24.097349    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43286820-8hgzf"]
Nov 02 23:04:24.683227 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:24.683197    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlpkd/must-gather-ng6vh" event={"ID":"a4ee7a8f-b582-44c8-9460-3e0466bcb3bd","Type":"ContainerStarted","Data":"7e1a40513ada1fb804fcab78bc7580c05395b1239623b6434b7b63931af7974e"}
Nov 02 23:04:24.912677 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:24.912643    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708ccd0d-118a-40fb-ba51-9ddc657cb131" path="/var/lib/kubelet/pods/708ccd0d-118a-40fb-ba51-9ddc657cb131/volumes"
Nov 02 23:04:25.690744 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:25.690701    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlpkd/must-gather-ng6vh" event={"ID":"a4ee7a8f-b582-44c8-9460-3e0466bcb3bd","Type":"ContainerStarted","Data":"0013b6320b2bb2001e8f2d59cfcbd87477a65116d590a29e68501ae07a79a4ee"}
Nov 02 23:04:25.805999 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:25.805937    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vlpkd/must-gather-ng6vh" podStartSLOduration=2.425473939 podStartE2EDuration="3.805917687s" totalImagesPullingTime="1.380443748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:25.804000142 +0000 UTC m=+6953.067536458" watchObservedRunningTime="2026-11-02 23:04:25.805917687 +0000 UTC m=+6953.069454008"
Nov 02 23:04:32.675138 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:32.674912    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zl2hk_b9be1c6f-dae7-418e-ae7d-d0cd42578469/global-pull-secret-syncer/0.log"
Nov 02 23:04:34.327330 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:34.327295    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74879c7667-jgdxh_8ddc5d0a-c7a0-412a-ac84-c522dfcba78c/manager/0.log"
Nov 02 23:04:34.515573 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:34.515539    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9xjsn_4be9c715-baba-45d6-8cd9-19d679e10c2a/manager/0.log"
Nov 02 23:04:35.051588 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:35.051548    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"]
Nov 02 23:04:35.083133 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:35.082142    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qzbfv_8444a0ef-d9fe-4605-97e3-6d6263228a11/konnectivity-agent/0.log"
Nov 02 23:04:35.100277 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:35.098024    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f2aba-46ck9"]
Nov 02 23:04:35.887001 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:35.886970    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rgnpj_cb32ed85-0c0b-4b06-95c0-8cf300e337f5/server/0.log"
Nov 02 23:04:36.269221 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:36.269195    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-86.us-west-2.compute.internal_32f26124355cba5ec51ff80a5481f572/haproxy/0.log"
Nov 02 23:04:36.913129 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:36.913086    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d07466-7e06-4094-987e-a9ac0d9d3723" path="/var/lib/kubelet/pods/57d07466-7e06-4094-987e-a9ac0d9d3723/volumes"
Nov 02 23:04:37.403310 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:37.403282    4528 scope.go:122] "RemoveContainer" containerID="8f198949d46f099539d9ab92d1949c9ece2fb994d340e271548e0ff991b907c7"
Nov 02 23:04:37.421342 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:37.421318    4528 scope.go:122] "RemoveContainer" containerID="a150ebc1633edbcdc56484d0010ea722075b9930c3ef71aabdf4b0e567ee8094"
Nov 02 23:04:39.986255 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:39.986218    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74879c7667-jgdxh_8ddc5d0a-c7a0-412a-ac84-c522dfcba78c/manager/0.log"
Nov 02 23:04:40.177924 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:40.177891    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9xjsn_4be9c715-baba-45d6-8cd9-19d679e10c2a/manager/0.log"
Nov 02 23:04:40.884139 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:40.881991    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867cc8b678-sgzth_774878d1-3662-465a-8a44-3fa542eefd66/authorino/0.log"
Nov 02 23:04:41.632713 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.632678    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"]
Nov 02 23:04:41.638277 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.638251    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.691244 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.688503    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"]
Nov 02 23:04:41.691244 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.688733    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rgnpj_cb32ed85-0c0b-4b06-95c0-8cf300e337f5/server/0.log"
Nov 02 23:04:41.802088 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.802053    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-lib-modules\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.802320 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.802103    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dg79\" (UniqueName: \"kubernetes.io/projected/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-kube-api-access-8dg79\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.802320 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.802147    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-sys\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.802320 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.802279    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-proc\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.802320 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.802306    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-podres\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.903840 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.903764    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dg79\" (UniqueName: \"kubernetes.io/projected/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-kube-api-access-8dg79\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.903840 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.903800    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-sys\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.904102 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.903886    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-sys\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.904102 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.903941    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-proc\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.904102 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.903984    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-podres\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.904102 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.903983    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-proc\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.904102 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.904024    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-podres\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.904358 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.904134    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-lib-modules\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.904358 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.904238    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-lib-modules\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:41.960983 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:41.960952    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dg79\" (UniqueName: \"kubernetes.io/projected/15420c00-65b2-4f5b-9dd7-fa4fbec762e1-kube-api-access-8dg79\") pod \"perf-node-gather-daemonset-whd2r\" (UID: \"15420c00-65b2-4f5b-9dd7-fa4fbec762e1\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:42.251927 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:42.251841    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:42.629047 ip-10-0-133-86 kubenswrapper[4528]: W1102 23:04:42.629012    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod15420c00_65b2_4f5b_9dd7_fa4fbec762e1.slice/crio-5515d2da8dde00f17b41c9fbd922eac6fdfab2a1b4fe56ba859c662374ff6e01 WatchSource:0}: Error finding container 5515d2da8dde00f17b41c9fbd922eac6fdfab2a1b4fe56ba859c662374ff6e01: Status 404 returned error can't find the container with id 5515d2da8dde00f17b41c9fbd922eac6fdfab2a1b4fe56ba859c662374ff6e01
Nov 02 23:04:42.631187 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:42.631162    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"]
Nov 02 23:04:42.818472 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:42.818438    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r" event={"ID":"15420c00-65b2-4f5b-9dd7-fa4fbec762e1","Type":"ContainerStarted","Data":"f83c8e81f1aa5ec140d80f6fa50f560653995baf9d53a81e120094cb7c9d14b3"}
Nov 02 23:04:42.818929 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:42.818535    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:42.818929 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:42.818556    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r" event={"ID":"15420c00-65b2-4f5b-9dd7-fa4fbec762e1","Type":"ContainerStarted","Data":"5515d2da8dde00f17b41c9fbd922eac6fdfab2a1b4fe56ba859c662374ff6e01"}
Nov 02 23:04:42.933046 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:42.932942    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r" podStartSLOduration=1.932916082 podStartE2EDuration="1.932916082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 23:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:42.930483858 +0000 UTC m=+6970.194020188" watchObservedRunningTime="2026-11-02 23:04:42.932916082 +0000 UTC m=+6970.196452401"
Nov 02 23:04:44.032222 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:44.032191    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5tp5v_598f64cf-7fb5-4b62-a4ee-29802c83c4f4/limitador/0.log"
Nov 02 23:04:44.113256 ip-10-0-133-86 kubenswrapper[4528]: E1102 23:04:44.113220    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctfx4" containerName="extractor"
Nov 02 23:04:45.774898 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:45.774865    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74879c7667-jgdxh_8ddc5d0a-c7a0-412a-ac84-c522dfcba78c/manager/0.log"
Nov 02 23:04:45.970859 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:45.970813    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9xjsn_4be9c715-baba-45d6-8cd9-19d679e10c2a/manager/0.log"
Nov 02 23:04:47.223306 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:47.223278    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-rgnpj_cb32ed85-0c0b-4b06-95c0-8cf300e337f5/server/0.log"
Nov 02 23:04:48.835663 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:48.835635    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-whd2r"
Nov 02 23:04:53.513695 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:53.513657    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-n5c5h_c167e9c0-18fb-447d-b5ce-febde46c229f/discovery/0.log"
Nov 02 23:04:53.833140 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:53.833031    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78fbf7b575-zlh7p_cb4c26f6-5512-4b9d-9c66-0a2a45798eb3/kube-auth-proxy/0.log"
Nov 02 23:04:54.884832 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:54.884802    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-gj9qp_2a0bbf59-4728-4a9a-882a-44eec7d799df/istio-proxy/0.log"
Nov 02 23:04:54.947559 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:54.947527    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swfcc_13c612c3-8753-423a-b670-9ed1ccb628e6/node-exporter/0.log"
Nov 02 23:04:55.347718 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:55.347685    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swfcc_13c612c3-8753-423a-b670-9ed1ccb628e6/kube-rbac-proxy/0.log"
Nov 02 23:04:55.567318 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:55.567286    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-pwm9l_00ba69b8-9f85-4cd0-bad9-b8bc4bd752fd/istio-proxy/0.log"
Nov 02 23:04:55.749711 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:55.749637    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swfcc_13c612c3-8753-423a-b670-9ed1ccb628e6/init-textfile/0.log"
Nov 02 23:04:55.779347 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:55.779318    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-tgp7n_5ddfda63-2274-4793-bb2a-e99d52000804/istio-proxy/0.log"
Nov 02 23:04:56.407769 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:56.407743    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7669ccf7ff-28hfp_cc1bf636-d125-4f0c-b76f-6a11c1f26457/payload-pre-processing/0.log"
Nov 02 23:04:56.617620 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:04:56.617588    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6cd6f7444d-c7gdn_d0d9de36-b748-4d05-9e97-2182dbed6f6b/payload-processing/0.log"
Nov 02 23:05:00.212023 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:05:00.211994    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rk9ql_82467e24-4ccf-403d-915d-8b99419e3830/prometheus-operator/0.log"
Nov 02 23:05:00.611721 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:05:00.611690    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rk9ql_82467e24-4ccf-403d-915d-8b99419e3830/kube-rbac-proxy/0.log"
Nov 02 23:05:01.025021 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:05:01.024992    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-zj6gn_f8fbd01c-b980-4c76-825d-acae667c2904/prometheus-operator-admission-webhook/0.log"
Nov 02 23:05:02.055354 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:05:02.055319    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867cc8b678-sgzth_774878d1-3662-465a-8a44-3fa542eefd66/authorino/0.log"
Nov 02 23:05:03.310746 ip-10-0-133-86 kubenswrapper[4528]: I1102 23:05:03.310720    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5tp5v_598f64cf-7fb5-4b62-a4ee-29802c83c4f4/limitador/0.log"